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EXECUTIVE  SUMMARY 


Surface  combatants  are  one  of  the  most  valuable  assets  of  a  country.  They 
provide  security  on  the  sea  and  protect  a  nation’s  interests  across  the  globe.  For 
these  reasons,  surface  combatants  need  to  be  not  only  capable,  but  also  flexible 
and  ready  to  fight  any  threat  in  this  rapidly  changing  world. 

Modern  maritime  menaces  are  threatening  the  shipping  lanes  and 
merchant  vessels  as  well  as  the  warships.  Operations  such  as  naval  escort,  anti¬ 
piracy,  and  maritime  interdiction  have  begun  to  take  the  place  of  conventional 
naval  warfare.  However,  existing  ships  were  not  specifically  built  for  these 
modern  tasks,  and  they  are  extremely  expensive  to  risk  in  such  missions. 
Moreover,  they  are  not  flexible  enough  to  deal  with  small  targets.  Therefore,  most 
navies  have  started  building  smaller,  more  flexible  ships  to  cope  with  evolving 
maritime  threats. 

Assembling  a  modern,  capable,  and  flexible  fleet  is  challenging.  The 
acquisition  of  new  naval  ships  requires  thorough  analysis  of  the  potential  vessel’s 
capability,  cost,  and  operational  effectiveness.  NPS  is  a  participant  in  an  Office  of 
Naval  Research  (ONR)  initiated  project  that  focuses  on  the  decisions  involved  in 
the  early  stages  of  the  shipbuilding  process.  The  NPS  Operations  Research  and 
Systems  Engineering  departments  are  working  together  to  form  a  decision¬ 
making  tool,  called  the  NPS  Dashboard,  that  demonstrates  the  trade-space 
among  capability,  cost,  and  operational  effectiveness  of  the  naval  ships.  The 
purpose  of  this  thesis  is  to  help  NPS  integrate  the  operational  effectiveness  of  an 
Offshore  Patrol  Vessel  (OPV)  within  an  anti-surface  warfare  (ASUW) 
environment  using  the  NPS  Dashboard. 

Analyzing  the  operational  effectiveness  of  a  ship  requires  a  scenario  to 
build  a  combat  model  and  a  modeling  platform  to  run  multiple  simulations.  The 
scenario  studied  in  this  research  is  based  on  an  incident  that  occurred  2002.  In 
the  scenario,  the  terrorists  are  planning  to  perform  a  suicide  attack  on  a  high 
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value  unit  (HVU)  in  the  Strait  of  Gibraltar.  The  mission  of  the  OPV  is  to  escort  the 
HVU  while  crossing  the  strait.  If  the  HVU  can  pass  through  the  strait,  then  it  is 
assumed  to  be  a  mission  success;  otherwise,  if  it  gets  hit,  the  mission  is 
assumed  to  be  a  failure.  This  scenario  is  instantiated  in  an  agent-based  combat 
modeling  platform  called  Map  Aware  Non-Uniform  Automata  (MANA).  MANA 
provides  powerful  tools  for  realistically  representing  the  world  in  a  simulation 
environment.  A  screenshot  of  the  combat  model  used  for  this  research  is  shown 
below. 


Helicopter  Unknown  Vessel 


/  Known  Vessel 


Hostile  Boat 


HVU 
OPV 


Strait  of  Gibraltar 


There  are  six  types  of  vessels  in  the  scenario:  HVU  (pink),  OPV  (blue),  helicopter 
(blue),  hostile  boats  (red),  known  vessels  (green),  unknown  vessels  (yellow). 


A  Design  of  Experiments  (DoE)  approach  is  used  for  varying  controllable 
and  uncontrollable  factors  that  are  affecting  the  outcome  of  the  mission. 
Controllable  factors  include  the  capability,  tactics,  and  rules  of  engagement 
(ROE)  that  are  related  to  the  decisions  about  friendly  assets.  Uncontrollable 
factors  are  related  to  hostile  boats,  known  vessels,  and  unknown  vessels.  The 
data  analysis  is  done  using  regression  tools  and  partition  trees.  The  results  of  the 
analysis  show  that  both  capability  choices  and  tactical  decisions  have  substantial 
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effects  on  the  result  of  a  naval  mission.  Several  of  the  key  drivers  of  mission 
success  in  this  scenario  are  shown  below. 


Factors 

Capability 

Tactics 

ROE 

Main  Gun  Presence 

X 

X 

X 

Leading  Ship  Choice 

X 

Range  to  Start  Counter  Measures 

X 

X 

X 

Auxiliary  Gun  Flit  Probability 

X 

X 

FHelicopter  Presence 

X 

X 

X 

OPV  Maximum  Speed 

X 

Key  decision  factors  that  affect  mission  success. 


Most  decisions  about  a  naval  ship’s  capabilities  are  made  prior  to  the 
design  of  the  ship.  For  better  decisions,  the  decision  maker  needs  to  know  the 
trade-offs  between  certain  options.  The  analysis  for  the  operational  effectiveness 
of  the  ships  is  therefore  crucial.  The  analyst  helps  the  decision  maker  choose  the 
best  available  ship  design  among  the  alternatives.  Furthermore,  the  tactical 
decisions  of  the  naval  officers  are  also  paramount  for  accomplishing  a  mission. 
This  thesis  shows  how  both  the  capability  and  operational  decisions  can  be 
analyzed  in  the  early  stages  of  the  ship  design.  Even  the  concept  of  operations 
(CONOPS)  for  various  tasks  can  be  planned  along  with  the  ship  design  process. 

In  the  past,  the  experience  of  the  sailors  was  the  main  driver  of  the  tactical 
decisions.  Today,  it  is  the  technology  and  the  science  that  shape  tactical  decision 
making  more  intensely.  Naval  officers  should  know  the  characteristics  of  their 
ships,  and  they  should  rely  not  only  their  experience  but  also  on  analytical  tools 
to  form  their  decisions.  Choosing  the  right  course  of  action  may  reduce  the 
emphasis  on  capability  as  the  primary  factor  in  decision  making.  This  change  has 
the  potential  to  reduce  the  acquisition  cost  of  a  naval  ship  while  improving  the 
navy’s  prospects  for  achieving  mission  success. 
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I.  INTRODUCTION 


“He  who  rules  the  seas,  rules  the  world” 

-  Admiral  Barbaras  Hayrettin  Pasha 


A.  OVERVIEW 

In  the  past  two  decades,  there  has  been  a  significant  shift  in  naval 
missions  toward  operations  other  than  war.  Counter-piracy,  search  and  rescue, 
maritime  interdiction,  maritime  patrol,  and  naval  escort  operations  are  the  main 
focus  of  most  fleets  today;  however,  the  vessels  that  are  currently  being  used  in 
such  operations  were  mainly  built  for  other  purposes.  For  instance,  in  August 
2009,  the  North  Atlantic  Council  approved  “Operation  Ocean  Shield”  to  fight 
piracy  in  the  Gulf  of  Aden.  Among  six  surface  ships  that  were  assigned  in  the 
January-June  2012  rotation  of  this  NATO  mission,  one  was  a  destroyer  and  three 
were  frigates  (“NATO  -  Counter-piracy  operations,”  n.d.).  Although  those 
warships  can  be  used  in  such  missions,  how  reasonable  it  is  to  risk  a  destroyer 
or  a  frigate  to  fight  with  terrorist  boats  or  pirates? 


* 
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Figure  1 .  NATO  warships  in  SNMG2  operation  (image  from  Allied  Maritime 
Command  Naples  -  NATO,  http://www.manp.nato.int) 
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Many  surface  vessels  that  perform  these  modern  tasks,  as  in  the  NATO  Task 
Force  example,  are  sophisticated  warships  capable  of  anti-surface  warfare 
(ASUW),  anti-air  warfare  (AAW),  and  anti-submarine  warfare  (ASW).  Although 
these  sophisticated  multi-mission  capable  fleets  are  able  to  achieve  good  results 
in  expeditionary  warfare  against  a  strong  enemy  (Murphy,  2007),  the  capabilities 
of  those  ships  will  probably  be  used  in  less  than  1%  of  their  total  life  time.  It 
seems  a  sound  reason  to  build  capable  ships  in  case  of  a  conventional  war,  and 
one  can  claim  that  capable  ships  are  built  to  be  used  in  that  small  period  of  their 
life  time;  nevertheless,  navies  should  optimize  their  efforts  and  resources  in 
some  way  to  find  a  better  mix  of  vessel  types  and  a  better  mix  of  systems  that 
constitute  the  vessels. 

While  frigates  and  destroyers  seem  to  be  the  best  available  options  that  can 
perform  offshore  operations,  they  are  expensive  to  build  and  operate.  On  the 
other  hand,  smaller  combatants  are  much  cheaper  and  better  suited  for  modern 
naval  operations  due  to  their  flexibility.  Therefore,  many  nations  have  begun 
reshaping  their  fleets  to  meet  emerging  operational  demands:  (1)  they  have 
started  building  smaller,  yet  sophisticated  multi-mission  capable  combatants 
such  as  Offshore  Patrol  Vessels  (OPVs)  and  Littoral  Combat  Ships  (LCSs)  that 
can  be  operated  in  a  variety  of  maritime  roles;  (2)  they  have  initiated  the 
development  of  new  tactics  and  counter  measures  to  better  deal  with  the  new 
threats.  The  ship  design  and  the  acquisition  processes,  however,  remains  almost 
the  same. 

Despite  technological  advances  in  the  last  few  decades,  the  ship  design 
and  the  acquisition  processes  have  not  yet  been  able  to  keep  pace  with  the  rapid 
changes  in  use  of  technology  (Ryan  &  Jons,  1992).  Cost  effectiveness  and 
operational  effectiveness  are  important,  and  it  is  extremely  hard  to  achieve  both 
using  a  traditional  ship  design  process.  Acceptable  levels  of  effectiveness  for 
both  measures  are  more  likely  to  be  achieved  with  the  use  of  technology  and 
virtual  environments.  Moreover,  utilizing  the  simulations  and  analytical  models  to 
build  decision  making  tools  will  ensure  collaboration  between  warfighters  and 
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engineers  in  the  early  stages  of  the  process.  Therefore,  exploiting  technology  is 
paramount  for  accomplishing  a  navy’s  objectives  and  increasing  the 
effectiveness  for  both  cost  and  operations  (Mizine,  Wintersteen,  &  Wynn,  2012). 

B.  BACKGROUND 

In  December  2010,  Orizzonte  Sistemi  Navali  (OSN),  an  Italian  contractor 
and  whole  warship  design  authority,  and  the  Office  of  Naval  Research  (ONR), 
initiated  a  project  called  “the  Application  System  for  Naval  Evaluation  and  Testing 
(ASNET),  Partnership  for  Research  on  Naval  Technology  and  Operations 
(PRONTO),  Naval  International  Cooperative  Opportunities  in  Science  & 
Technology  Program  (NICOP).”  The  main  purpose  of  this  project  is  to  analyze 
the  operational  effectiveness  of  ships  using  simulation  and  analytical  models,  to 
create  a  ship  synthesis  model,  and  most  importantly,  to  develop  decision  making 
tools  for  ship  designs  using  a  model-based  ship  design  approach.  These  tools 
will  ensure  that  decision  makers  gain  useful  insight  by  exploring  the  trade  space 
between  factors  affecting  ship  design  and  make  informed  decisions  (Perra, 
Guagnano,  &  Bonvicini,  2012). 

The  model-based  ship  design  approach  used  in  this  project  is 
fundamentally  different  from  the  way  surface  combatants  have  been  built  in  the 
past.  Previously,  the  focus  has  been  on  engineering  design  criteria;  mission 
types  and  operational  environments  have  not  been  considered  until  after  the  ship 
design  is  essentially  complete.  Nevertheless,  the  model-based  approach  ensures 
that  the  mission  requirements  are  linked  to  the  capability  analysis  during  the  ship 
production  process.  Using  this  method,  which  is  essentially  the  application  of 
Model-Based  Systems  Engineering  (MBSE)  to  decision  making,  helps  ensure 
that  the  Navy’s  needs  are  better  translated  into  ship  requirements,  and  the 
decision  makers  get  what  they  really  need  to  get  at  the  end  of  the  process 
(Robinson,  Tramoundanis,  Harvey,  Jones,  &  Wilson,  2010). 

The  chosen  subject  of  the  ASNET  PRONTO  NICOP  project  is  OPVs; 
therefore,  the  scenarios,  the  cost  data,  and  capability  analyses  are  based  on 
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OPV  characteristics.  The  Naval  Postgraduate  School  (NPS),  Georgia  Institute  of 
Technology,  University  of  Genoa,  and  some  other  institutions  are  participating  in 
this  effort.  The  role  of  NPS  is  important  in  the  project.  The  Operations  Research 
(OR)  and  Systems  Engineering  (SE)  departments  are  collaborating  to  create  a 
dashboard  that  displays  cost,  capability,  and  operational  trade-offs  (Figure  2). 
The  operational  experience  of  the  students  and  the  faculty  members  is  leveraged 
especially  in  building  scenarios  and  simulation  models.  The  insights  gained  from 
this  work  are  frequently  shared  with  ONR,  OSN,  and  other  participants  during  the 
workshops  and  the  meetings. 
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Figure  2.  NPS  Dashboard  showing  linkage  of  operational  and  physical  trade- 

space. 
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As  shown  in  Figure  3,  three  main  categories  of  data  are  fed  into  forming 
the  ship  synthesis  dashboard: 

•  Operational  Effectiveness 

•  Capability 

•  Cost 

The  operational  effectiveness  data  and  the  cost  estimation  data  are  mostly 
provided  by  the  OR  students,  whereas  the  capability  data  are  provided  by 
students  in  the  SE  curriculum. 


Figure  3.  A  sample  list  of  Measures  of  Effectiveness  (MOEs)  and  Measures  of 
Performance  (MOPs)  to  include  in  the  dashboard. 


Until  now,  several  OR  students  built  ASUW,  Search  and  Rescue,  and 
Maritime  Interdiction  Operation  scenarios  to  evaluate  key  performance  factors  for 
ship  designs.  In  their  theses,  their  primary  goal  was  to  establish 
interchangeability  with  the  models  OSN  created,  which  were  simple  and 
unrealistic.  Moreover,  those  theses  made  limited  comments  on  how  tactics  affect 
the  results.  Therefore,  more  realistic  and  advanced  scenarios  need  to  be  created 
to  be  able  to  develop  a  truly  useful  dashboard.  The  factors  that  affect  the  ship 
design  needs  to  be  reconsidered  as  well. 
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C.  RESEARCH  QUESTIONS 

This  research  detailed  in  this  thesis  focuses  on  the  following  questions 
related  to  the  improvement  of  the  ship  design  process: 

•  Based  on  a  realistic  and  an  advanced  ASUW  scenario,  which  ship  design 
factors  are  key  drivers  of  OPV  performance  in  ASUW? 

•  What  are  the  strengths  and  limitations  of  utilizing  Map  Aware  Non-Uniform 
Automata  -  Vector  (MANA-V),  an  agent-based  modeling  platform 
developed  by  New  Zealand’s  Defense  Technology  Agency  (DTA),  to 
construct  advanced  and  realistic  scenarios  for  evaluating  effectiveness  of 
naval  ships? 

•  How  do  different  ship  capabilities,  tactics,  and  Rules  of  Engagement 
(ROE)  affect  the  mission  success? 

D.  SCOPE  OF  THE  THESIS 

A  recent  thesis  by  Jason  McKeown  analyzed  the  operational  effectiveness 
of  an  OPV  in  an  ASUW  environment  (McKeown,  2012).  His  model  was  based  on 
a  swarm  attack  scenario,  and  his  main  concern  was  to  establish 
interchangeability  with  OSN  model  as  mentioned  above.  In  his  scenario,  the 
hostile  boats  were  trying  to  reach  a  goal  line,  and  they  were  not  able  to  attack  the 
OPV. 

In  light  of  the  increase  in  maritime  terrorism,  it  is  also  reasonable  to  expect 
an  individual  boat  attack  on  a  friendly  ship  while  it  is  passing  through  a  strait, 
refueling  in  a  port,  or  patrolling  the  area.  The  main  focus  of  this  thesis  is, 
therefore,  assessing  the  mission  effectiveness  of  a  surface  combat  vessel  in  a 
realistic  ASUW  environment.  An  advanced  asymmetric  warfare  scenario  that  was 
not  previously  implemented  by  OSN  or  NPS  students  is  built  and  analyzed  as 
part  of  this  thesis.  The  factors  tested  are  more  realistic,  and  several  tactics,  such 
as  changing  the  course  of  the  ship  after  facing  an  attack,  are  analyzed  to  better 
understand  how  they  change  the  response  surface.  Additionally,  hostile  boats 
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are  smart  in  this  scenario.  For  instance,  they  are  able  to  commit  a  kamikaze 
attack  on  the  friendly  ships. 

E.  CHAPTER  OUTLINE 

Chapter  II  is  the  literature  review  that  touches  on  the  tools  used  in  this 
thesis,  the  naval  warfare  areas,  the  OPVs,  and  MBSE.  Brief  descriptions  based 
on  the  published  references  prepare  the  reader  for  the  following  chapters. 

Chapter  III  contains  the  development  of  the  combat  model  and  a  thorough 
description  of  the  operational  scenario  used  in  this  thesis.  The  types  of  ships 
used  in  the  combat  model  are  explained  in  this  chapter  as  well. 

Chapter  IV  covers  the  exploration  of  the  combat  model.  It  starts  with  the 
description  of  the  experimental  design  used  for  the  simulations  and  continues 
with  the  explanations  of  the  controllable  and  the  uncontrollable  factors.  Following 
this  discussion,  the  model  output  is  analyzed  using  several  types  of  data  analysis 
approaches,  including  regression  and  partition  tree  analysis.  Significant  factors 
are  discussed  along  with  the  operational  insights.  Other  implementations  of  the 
scenario  are  explained  as  well. 

Chapter  V  concludes  the  thesis.  It  gives  a  summary  of  the  study  and 
provides  recommendations  and  suggestions  for  further  research. 

F.  BENEFITS  OF  THIS  STUDY 

The  results  of  this  study  are  being  used  to  augment  the  ship  synthesis 
model  built  by  the  Systems  Engineering  department,  and  ultimately  provide 
ASNET  PRONTO  NICOP  project  participants  with  a  decision  making  tool  for 
naval  ship  design.  This  study  is  also  likely  to  help  decision  makers  obtain  a  better 
understanding  of  the  factors  effecting  ASUW  for  surface  combatants.  Since  the 
factors  examined  by  this  thesis  are  related — but  not  limited — to  OPVs,  the 
insights  obtained  from  this  study  can  potentially  be  used  for  other  ship  types, 
including  frigates,  fast  patrol  boats,  or  coast  guard  boats.  Additionally,  this  study 
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demonstrates  how  tactics  may  affect  mission  success  and  how  naval  officers 
should  use  their  firepower  and  maneuvering  capability  wisely. 
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II.  LITERATURE  REVIEW 


A.  ANTI-SURFACE  WARFARE  AND  MARITIME  TERRORISM 

There  are  three  primary  categories  of  naval  warfare:  surface,  air  and 
submarine  (Cole,  2007).  Among  those  three  warfare  areas,  surface  warfare  is 
perhaps  the  most  dominant  one,  mainly  because  other  areas  of  warfare  usually 
take  place  in  actual  war  conditions.  The  Department  of  Defense  Dictionary  of 
Military  and  Associated  Terms  describes  surface  warfare  as  “That  portion  of 
maritime  warfare  in  which  operations  are  conducted  to  destroy  or  neutralize 
enemy  naval  surface  forces  and  merchant  vessels”  ( DoD  Dictionary,  2006).  The 
term  anti-surface  warfare  (ASUW)  is  generally  used  in  a  similar  meaning,  and 
can  be  defined  as  operations  that  are  performed  against  surface  vessels  utilizing 
radar,  guns,  or  any  other  means  (Cole,  2007). 

The  reason  why  ASUW  is  the  most  common  type  of  warfare  lies  behind 
this  fact  that  ASUW  operations  not  only  include  fighting  with  enemy  state  surface 
vessels,  but  also  include  fighting  with  asymmetric  threats  such  as  small  suicide 
vessels  or  pirate  boats.  Naval  vessels  can  face  maritime  threats  at  any  time, 
since  the  terrorists  and  pirates  can  easily  purchase  and  possess  boats  and 
attack  a  merchant  vessel  or  a  warship;  therefore,  those  boats  cause  a  threat,  and 
they  must  be  neutralized  or  destroyed  whenever  necessary. 


9 


Figure  4.  A  helicopter  firing  warning  shots  in  front  of  the  pirate  boat  (image  from 
Allied  Command  Operations  -  Nato,  http://www.aco.nato.int) 


Even  though  the  term  “maritime  terrorism”  has  been  in  the  literature  for 
more  than  a  few  decades,  it  was  not  really  spelled  out  loudly  until  a  series  of 
incidents  started  in  2000.  In  January  2000,  terrorists  tried  to  attack  USS  The 
Sullivans  (DDG-68)  in  Yemen.  They  failed  because  the  terrorist  boat  sank  right 
before  the  attack.  Following  this  event,  the  terrorists  attacked  the  USS  Cole 
(DDG-67)  with  a  suicide  boat,  killing  17  of  the  crew  members,  in  October  2000. 
Almost  two  years  after  these  incidents,  in  October  2002,  a  boat  with  explosives 
hit  the  French  oil  tanker  Limburg  which  was  close  to  Yemen  coastal  waters  (Luft 
&  Korin,  2004).  These  are  just  a  few  examples  of  how,  with  just  a  small  boat, 
terrorists  can  cause  damage  to  multi-million/billion  dollar  ships  and — more 
importantly — kill  tens  of  innocent  people.  There  have  also  been  many  other 
terrorist  activities  which  were  prevented  in  their  planning  phases.  In  June  2002, 
for  instance,  Moroccan  officials  arrested  a  group  of  terrorists  suspected  of 
planning  an  attack  on  U.S.  and  British  merchant  vessels  when  passing  through 
the  Strait  of  Gibraltar  (Maggio,  2008).  These  incidents  show  that  maritime 
terrorism  is  a  problem  that  must  be  prevented  by  any  means,  and  the  surface 
combatants  must  be  ready  to  fight  the  terrorists  at  sea. 
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Figure  5.  USS  Cole  after  the  attack  in  October  2000  (image  from  New  York 

Times,  http://www.nytimes.com) 

B.  AGENT-BASED  MODELING  AND  MANA 

In  recent  years,  there  has  been  an  increase  in  the  use  of  agent-based 
modeling  and  simulation.  The  agent-based  modeling  approach  provides 
exceptional  tools  for  modelers  to  represent  the  real  world  within  a  computer 
program.  The  agent-based  models  consist  of  autonomous  entities  called  agents, 
and  the  set  of  rules  which  determine  the  interactions  between  agents  and  the 
environment  surrounding  them  (Bonabeau,  2002).  Although  there  is  not  a  precise 
definition  of  an  “agent”  (Macal  &  North,  2009)  or  “agent-based  simulation,”  in  the 
agent-based  modeling  concept,  the  agents  can  be  defined  as  the  representation 
of  the  real  world  objects  in  an  artificial  environment  (Cioppa,  Lucas,  &  Sanchez, 
2004).  The  agents  can  have  behaviors,  which  makes  the  agent-based  modeling 
technique  unique.  These  autonomous  entities  are  aware  of  the  events  that  they 
can  detect  by  organic/inorganic  sensors,  and  they  respond  to  the  environment 
with  actions  defined  by  algorithms. 
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There  are  many  tools  that  can  be  used  for  agent-based  modeling.  These 
include  general  purpose  programs  such  as  Microsoft  Excel  VBA,  C++,  JAVA; 
computation  and  statistics  programs  such  as  MATLAB,  Mathematica,  R;  and 
dedicated  agent-based  modeling  platforms  such  as  NetLogo,  Repast,  AnyLogic, 
MANA  (Macal  &  North,  2009).  Each  of  these  software  packages  has  some 
superiority  in  various  real  word  applications,  i.e.,  business  applications,  biological 
applications,  and  medical  applications;  nonetheless,  when  it  comes  to  military 
applications,  not  all  of  them  are  capable  of  modeling  the  non-linear  and  complex 
nature  of  the  military  conflicts  (Lauren  &  Stephen,  2002). 
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Figure  6.  NetLogo  is  a  widely  used  agent-based  modeling  platform  (image  from 
Science  Education  Resource  Center,  http://serc.carleton.edu) 


Military  conflicts  usually  exhibit  complexity  in  the  interaction  between  the 
agents  with  other  agents  and  the  environment.  Additionally,  agent  positions  in 
space  change  rapidly  throughout  time,  and  the  population  that  the  agents 
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represent  is  usually  heterogeneous  (Bonabeau,  2002).  All  these  attributes  show 
that  agent-based  modeling  is  a  proper  tool  to  represent  military  operations,  and 
choosing  the  right  agent-based  combat  modeling  platform  will  help  military 
analysts  to  better  represent  the  military  tasks  in  a  computer  program.  There  are 
several  agent-based  modeling  programs  that  can  be  practically  used  for 
modeling  operational  scenarios,  including  MANA,  Pythagoras,  and  JANUS 
(Cioppa  et  al.,  2004).  Among  these  programs,  MANA  stands  out  as  the  most 
user  friendly  platform,  and  it  is  has  been  widely  used  by  NPS  faculty  and 
students  for  almost  10  years.  Military  applications  of  agent-based  modeling  and 
simulation  done  using  MANA  include  maritime  protection  of  critical  infrastructure 
assets  (Lucas,  Sanchez,  Martinez,  Sickinger,  &  Roginski,  2007),  counter-piracy 
operations  (Tsilis,  2011),  UAV  operations  (Raffetto,  2004),  and  a  comparison  of 
warships  against  threats  in  confined  waters  (Ozdemir,  2009). 
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A  friendly  ship  heading  northward  through  a  region  with  two  hostile 
boats  on  patrol. 
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However,  it  tends  to  be  outnumbered. 
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Figure  7.  Screenshot  of  agent-based  combat  modeling  platform  MANA. 


Designed  by  the  Operational  Analysis  personnel  of  the  New  Zealand’s 

DTA,  MANA  is  an  agent-based  distillation  model  that  has  been  available  for  more 

than  ten  years  now  (Lauren  &  Stephen,  2002).  The  key  attributes  of  MANA  which 
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make  it  a  useful  tool  for  military  applications  are  the  situational  awareness  of  the 
agents,  advanced  communication  capabilities  within  squads  and  with  other 
agents,  the  interaction  of  entities  with  friends  and  foes,  and  the  user  friendly 
design  of  the  program. 

There  are  four  main  sets  of  parameters  that  form  agent  behaviors  in 
MANA  (McIntosh,  Galligan,  Anderson,  &  Lauren,  2007): 

•  Personality  weightings  determine  the  willingness  of  agents  to 
perform  a  particular  action. 

•  Movement  constraints  modify  the  basic  personality  weightings  of 
the  agents. 

•  Intrinsic  capabilities  determine  the  physical  characteristics  of  the 
agents  such  as  sensors,  weapons,  or  fuel  level. 

•  Movement  characteristic  adjustments  ensure  that  agent  actions 
change  in  different  terrain  conditions  and  different  situations. 
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Figure  8.  MANA-Screenshot  of  the  “Squad  Properties.” 


Just  recently,  DTA  released  the  new  version  of  MANA,  namely  MANA-V. 
“V”  stands  for  both  vector  and  five.  In  this  version,  the  programmers  replaced  the 
cell-based  movement  of  previous  versions  with  vector-based  movement,  which 
allows  building  larger  battlefield  regions  with  panning  and  zooming  options,  and 
allows  defining  the  distances  and  the  attributes  such  as  speed  and  range  in  real- 
world  units,  such  as  nautical  miles  (knots)  or  meters,  rather  than  pixels 
(McIntosh,  2009).  In  this  thesis,  MANA-V  version  5.01.04  is  used  for  modeling 
and  simulation. 

C.  DESIGN  OF  EXPERIMENTS 

In  the  real  world,  experiments  are  usually  built  varying  less  than  a  handful 
of  factors  or  variables.  Utilizing  the  power  of  computers,  on  the  other  hand,  it  is 
possible  to  build  simulations  with  more  variables  (Kleijnen,  Sanchez,  Lucas,  & 
Cioppa,  2005).  However,  increasing  the  number  of  variables  often  causes  high 
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runtimes  despite  the  modern  computers.  Even  with  the  technological  advances 
and  supercomputers,  it  may  take  months  to  run  a  simulation  experiment  with  a 
large  number  of  factors  using  a  brute  force  approach.  Utilizing  design  of 
experiments  (DOE),  coupled  with  the  powerful  computers,  solves  this  problem. 
DOE  makes  it  possible  to  build  simulations  with  a  relatively  high  number  of 
parameters,  and  it  gives  an  acceptable  screening  property  of  the  response 
surface,  which  leads  to  gaining  better  insights  from  the  experiment. 

The  experimental  design  approach  allows  the  analyst  to  provide  more 
information  to  decision  makers  in  a  relatively  small  amount  of  time  and  helps 
them  better  understand  the  factors  affecting  the  results  (Kleijnen  et  al.,  2005). 
Instead  of  changing  factors  one  at  a  time  or  choosing  design  points  with  trial- 
error,  DOE  establishes  smart  ways  of  designing  an  experiment,  which  will  yield 
important  insights  both  to  the  analyst  and  the  decision  makers  (Sanchez,  2007). 
In  addition  to  exploring  more  factors,  DOE  also  helps  the  analyst  spend  less  time 
on  the  simulation  runs  and  more  time  on  the  analysis. 

There  are  several  DOE  techniques  in  the  literature.  The  ones  specifically 
useful  in  simulation  design  include  factorial  designs,  fractional  factorial  designs, 
central  composite  designs,  and  Latin  hypercube  (LH)  designs.  Each  of  these 
techniques  has  some  advantages  and  disadvantages  depending  on  the  number 
of  factors,  screening  capabilities,  and  response  surface  complexity  (Figure  9). 
For  instance,  with  only  a  few  factors,  coarse  grids  might  be  useful.  On  the  other 
hand,  if  the  number  of  the  factors  is  very  large,  LH  designs  are  more  efficient 
than  the  other  designs,  since  they  provide  well  screening  of  the  response  surface 
and  yield  similar  information  with  significantly  less  computational  effort  (Sanchez, 
2007). 
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Figure  9.  Recommended  designs  according  to  the  number  of  factors  and  system 
complexity  assumptions  (Adapted  from  Kleijnen  et  ai.,  2005) 


A  nearly  orthogonal,  nearly  balanced  (NOB)  mixed  design  approach  is 
used  to  build  the  experiment  in  this  thesis  (Vieira,  Sanchez,  Kienitz,  &  Belderrain, 
2011).  The  main  advantage  of  this  technique  is  that  it  accommodates  not  only 
continuous  factors,  but  also  discrete  and  categorical  factors  in  the  design. 

D.  OFFSHORE  PATROL  VESSEL  (OPV) 

Since  the  industrial  revolution,  surface  combatants  have  evolved  rapidly. 
Naval  shipyards  are  now  able  to  build  multi-purpose  warships  such  as  cruisers, 
destroyers,  and  frigates,  which  are  the  most  powerful  naval  assets  of  a  country. 
These  ships  can  operate  anywhere  globally  with  less  dependence  on  coastal 
facilities  or  resources.  However,  they  are  not  the  only  fighting  ships  that  a  navy 
has.  There  is  a  variety  of  fighting  ships,  ranging  from  small  patrol  boats  to 
cruisers,  each  of  which  has  advantages  in  different  situations.  For  instance,  small 
patrol  boats  have  a  small  draft,  which  allows  them  to  navigate  through  shallow 
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waters  close  to  the  shore.  On  the  other  hand,  larger  ships  are  able  to  perform 
multiple  tasks  simultaneously.  They  can  deal  with  aircraft,  surface  ships, 
missiles,  and  submarines  at  the  same  time,  and  they  can  perform  counter¬ 
measures  against  a  variety  of  enemy  forces. 

OPVs  are  relatively  small  combatants  when  compared  to  frigates  and 
destroyers.  They  are  more  like  corvettes,  with  less  fighting  capability.  Their  main 
purpose  is  maintaining  maritime  security  inshore  and  offshore,  and  they  can  be 
deployed  globally  (Kimber  &  Booth,  2010).  Many  countries  do  not  really  need 
sophisticated  naval  combatants,  and  some  countries  cannot  afford  capable  ships 
(Annati,  2009).  Being  cheaper  yet  flexible,  OPVs  are  great  options  for  those 
countries  throughout  the  world. 

Nations  build  OPVs  mainly  to  use  in  peacetime  operations,  and  the 
primary  functions  of  the  OPVs  are  ASUW  and  AAW;  however,  they  can  also  be 
used  in  ASW  and  other  warfare  areas,  if  built  with  such  capability  (Eames,  1985). 
Escort  operations,  anti-piracy  operations,  and  humanitarian  efforts  are  some  of 
areas  where  OPVs  are  used. 

E.  MODEL-BASED  SYSTEMS  ENGINEERING  (MBSE) 

Technological  advances  led  humans  to  build  complex  systems  over  the 
years.  While  the  systems  are  becoming  more  and  more  complex,  it  is  becoming 
extremely  hard  for  engineers  and  decision  makers  to  deal  with  development  of 
these  diverse  systems.  Surface  combatants  are  examples  of  such  complex 
systems.  They  consist  of  various  subsystems  such  as  weapons  and  sensors. 
Moreover,  the  stakeholders  or  shareholders  that  work  on  the  ship  building 
process  come  from  a  wide  range  of  industries  (Calvano,  Jons,  &  Keane,  2000). 
These  make  the  shipbuilding  process  extremely  complicated,  especially  if  the 
warship  is  the  first  one  of  its  class. 

Systems  engineering  tries  to  find  a  way  to  deal  with  the  complex  systems. 
According  to  Kossiakoff  et  al.  (2011),  the  function  of  systems  engineering  is 
“guiding  the  engineering  of  the  complex  systems.”  Furthermore,  it  is  argued  that 
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the  systems  engineering  effort  is  centered  on  the  systems  as  whole,  not  on  the 
separate  parts  (Kossiakoff,  Sweet,  Seymour,  &  Biemer,  2011). 

Model-based  systems  engineering  (MBSE)  is  a  new  approach  to  systems 
engineering,  and  it  is  seen  as  the  future  of  the  systems  engineering  by 
International  Council  on  Systems  Engineering  (INCOSE).  Systems  Engineering 
Vision  2020  describes  MBSE  as  “the  formalized  application  of  modeling  to 
support  system  requirements,  design,  analysis,  verification  and  validation 
activities  beginning  in  the  conceptual  design  phase  and  continuing  throughout 
development  and  later  life  cycle  phases”  (Crisp,  2007).  This  approach  is  mainly 
useful  for  highly  complex  systems  such  as  warships  and  maintains  the  link 
between  the  engineers  and  the  decision  makers  (Robinson  et  al.,  201 0). 
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III.  MODEL  DEVELOPMENT 


A.  BACKGROUND 

Operations  to  fight  terrorist  boats  or  pirate  ships  have  become  a  crucial 
aspect  of  naval  operations  since  terrorists  have  started  being  a  threat  to  military 
and  civilian  ships  in  the  maritime  domain.  In  the  last  15  years,  there  has  been  a 
significant  increase  in  the  number  of  incidents  in  which  terrorists  attacked  a 
warship  or  a  merchant  vessel  and  killed  tens  of  sailors.  Additionally,  there  were 
some  incidents  in  which  the  terrorists  were  not  able  to  carry  out  their  outrageous 
plans.  For  instance,  in  June  2002  a  group  of  terrorists,  who  were  planning  an 
attack  on  two  merchant  vessels  in  the  Strait  of  Gibraltar,  were  caught  by 
Moroccan  officials  (Maggio,  2008). 


Figure  1 0.  There  are  various  types  of  ships  passing  the  Strait  of  Gibraltar  at  any 
moment  of  a  day  (from  http://www.marinetraffic.com) 


The  fact  that  the  officials  arrested  those  terrorists  does  not  mean  that 
similar  plans  will  not  be  put  into  practice  by  other  groups.  If  a  terrorist  group  fills  a 
boat  with  explosives  and  approaches  a  ship  in  Strait  of  Gibraltar,  it  will  be  really 
difficult  to  identify  that  boat  as  a  terrorist  boat  since  there  is  large  number  of 
vessels  in  the  strait  at  any  moment  (Figure  10).  Therefore,  naval  escort  ships 
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need  to  be  ready  to  protect  both  themselves  and  the  merchant  vessels  against 
asymmetric  threats.  Most  current  naval  vessels,  however,  are  not  particularly 
built  to  fight  terrorist  boats,  and  they  are  expensive  to  risk  in  such  missions.  As  a 
consequence,  many  countries  started  building  smaller  yet  flexible  surface 
combatants,  such  as  OPVs,  to  meet  the  demands  of  their  navies.  Nevertheless, 
finding  the  best  mix  of  capabilities  such  as  weapons  and  maximum  speed  is  a 
problem  that  needs  to  be  considered  in  the  very  beginning  of  the  shipbuilding 
process  (Mizine  et  at.,  2012).  A  proper  way  to  overcome  this  problem  is  building 
combat  models  and  testing  the  operational  effectiveness  of  the  surface 
combatants  in  scenario-based  simulation  environments. 

B.  SCENARIO  DESCRIPTION 

In  this  thesis,  the  agent-based  modeling  platform  MANA  is  used  to  model 
a  naval  escort  mission  scenario  in  an  anti-surface  warfare  environment.  The 
scenario  is  based  on  the  Morocco  incident  that  occurred  in  2002.  There  are  six 
types  of  agents  in  the  scenario:  the  high  value  unit  (HVU)  being  escorted,  the 
OPV,  the  OPV’s  helicopter,  terrorist  boats,  known  vessels,  and  unknown  vessels. 
A  screenshot  from  MANA  appears  in  Figure  1 1 ;  note  that  the  sizes  of  the  agents 
are  not  to  scale,  but  are  magnified  for  easier  visualization  by  the  user.  The 
allegiance  of  the  HVU,  OPV,  and  the  helicopter  is  “friend,”  the  allegiance  of  the 
terrorist  boat  is  “hostile,”  and  the  allegiance  of  the  known  and  unknown  vessels  is 
“neutral.” 
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Figure  1 1 .  Screenshot  of  MANA.  There  are  six  types  of  agents  in  the  ASUW 
scenario:  HVU  (pink),  OPV  (blue),  helicopter  (blue),  terrorist  boats  (red), 
known  vessels  (green),  unknown  vessels  (yellow). 


1.  Friendly  Assets 

The  mission  of  the  friendly  assets  is  to  cross  the  Strait  of  Gibraltar  without 
allowing  any  damage  to  be  caused  to  the  HVU  by  hostile  vessels.  The  OPV  is 
tasked  to  protect  the  HVU  from  attacks  that  can  occur  in  the  passage.  It  has 
several  guns,  ranging  from  high  caliber  guns  to  machine  guns.  It  also  has  a 
helicopter  landing  platform.  The  main  role  of  the  helicopter  is  to  detect  and 
classify  unknown  vessels.  Its  high  speed  and  maneuverability  give  the  friendly 
forces  an  advantage  against  the  hostiles.  The  helicopter  can  also  have  a 
machine  gun  so  that  it  can  start  firing  before  the  hostile  vessels  come  closer  to 
the  HVU. 

2.  Hostile  Boats 

The  terrorist  boats  are  loaded  with  explosives,  and  their  purpose  is  to  get 

closer  to  a  high  value  unit  (HVU)  so  that  they  can  perform  a  suicide  attack.  The 

only  target  of  the  hostile  boats  is  the  HVU.  They  do  not  attack  the  OPV,  but  in 

some  cases  they  may  try  to  evade  it.  One  of  the  critical  properties  of  the  terrorist 
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boats  is  that  they  are  initially  acquired  as  unknown  vessels  in  the  friendly  force 
radar/systems  until  they  are  classified  as  enemy.  Therefore,  either  the  helicopter 
classifies  them  as  enemy,  or  the  unknown  vessels  get  closer  to  the  OPV  or  HVU, 
and  the  friendly  ships  classify  them  depending  on  the  range. 

3.  Neutral  Ships 

The  known  ships  and  the  unknown  ships  are  both  neutral,  and  they  do  not 
pose  a  threat  to  friendly  assets.  The  only  difference  between  the  two  is  that  while 
the  OPV  can  instantly  classify  the  known  neutral  ships  with  the  Automatic 
Identification  System  (AIS)  device,  unknown  ships  cannot  be  classified  when 
they  are  initially  detected.  This  occurs  for  several  reasons,  such  as  they  are  too 
small  to  carry  an  AIS  device  or  their  device  is  not  working.  Once  the  unknown 
vessels  are  classified,  they  become  known  vessels,  and  their  color  changes  to 
green  in  MANA.  The  importance  of  the  known  and  unknown  vessels  is  to 
represent  the  real  marine  traffic  of  the  Strait  of  Gibraltar  as  accurately  as 
possible.  When  the  neutral  ships  are  close  to  the  hostile  boats,  the  OPV  and  the 
aircraft  are  unable  to  fire  to  the  boats.  Moreover,  friendly  assets  need  to  put  in  an 
additional  effort  for  classifying  the  unknown  ships  as  neutral.  This  makes  the 
situation  more  complicated  and  gives  the  terrorists  a  chance  to  approach  at  a 
closer  distance  without  any  classification  by  the  friendly  force. 

C.  SCENARIO  ASSUMPTIONS  AND  LIMITATIONS 

The  scenario  is  based  on  a  real  incident;  nevertheless,  it  is  impossible  to 
create  a  model  that  imitates  the  real  world  exactly.  Therefore  some  assumptions 
need  to  be  made,  and  the  limitations  of  the  modeling  platforms  need  to  be 
considered  in  order  to  build  a  model  that  can  present  useful  insights. 

1.  Assumptions 

The  key  assumptions  used  to  build  the  model  are  the  following: 
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•  The  OPV  has  an  AIS  receiver,  which  helps  to  detect  and  classify 
the  neutral  ships  within  some  distance.  Once  they  are  classified, 
they  become  neutral  ships. 

•  Most  of  the  merchant  vessels  have  AIS  emitters.  They  provide  their 
data  to  the  friendly  forces.  However,  some  ships  do  not  have  AIS 
devices  even  though  they  are  neutral.  Those  ships  need  to  be 
detected  and  classified  visually  or  by  using  radar. 

•  The  terrorist  boats  can  communicate  with  each  other. 

•  The  helicopter  can  classify  the  unknowns  as  hostile  or  neutral  once 
it  approaches  within  a  certain  distance  to  the  target. 

•  The  terrorists  can  only  perform  a  kamikaze  attack.  They  do  not 
have  a  long  range  weapon  such  as  a  rocket. 

•  The  probability  of  a  hit  for  the  weapons  is  constant  within  the 
minimum  and  maximum  effective  ranges,  and  zero  outside  the 
range. 

2.  Limitations 

Most  of  the  limitations  are  related  to  the  combat  modeling  platforms.  For 
instance,  it  is  difficult  to  implement  a  naval  formation  using  different  squads; 
formation  is  only  allowed  for  agents  within  squads.  Another  limitation  is  that  it  is 
difficult  to  implement  tactics  in  the  model.  Nevertheless  there  are  ways  to 
overcome  many  such  limitations.  The  personalities  of  the  agents  in  MANA 
provide  exceptional  tools  for  implementing  tactics  and  behaviors  (Figure  12). 
Changing  the  levels  for  various  parameters  gives  the  modeler  an  opportunity  to 
make  the  agents  react  to  their  environment  in  a  smart  way. 
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Figure  1 2.  Personalities  of  the  agents  in  MANA. 


There  are  also  limitations  regarding  classified  information.  For  example, 
the  probability  of  hit  for  a  certain  weapon  is  classified  information,  and  it  is  not 
provided  in  any  open  source.  Moreover,  even  if  it  were  provided,  there  would  still 
be  some  uncertainties  about  the  probability  of  hit  depending  on  the  size  of  the 
target,  weather  conditions,  etc.  To  overcome  these  types  of  limitations,  properties 
of  the  agents  having  uncertainties  are  built  as  a  factor  in  the  DoE.  By  doing  that, 
a  wide  range  of  levels  for  each  property  is  tested.  Moreover,  this  will  allow  the 
analyst  with  access  to  the  classified  information  to  pull  out  the  setting  of  interest 
and  come  up  with  a  proper  result. 
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D.  MEASURE  OF  EFFECTIVENESS 

The  only  measure  of  effectiveness  in  the  scenario  is  the  probability  of 
mission  success.  If  the  HVU  can  cross  the  Strait  of  Gibraltar  and  reach  the  goal 
line,  it  is  a  mission  success.  If  at  least  one  of  the  terrorist  boats  is  able  to  attack 
the  HVU,  then  it  is  a  mission  failure. 
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IV.  MODEL  EXPLORATION 


A.  DESIGN  OF  EXPERIMENTS 

Hundreds  of  factors  affect  the  outcome  of  a  military  mission  in  the  real 
world.  These  factors  can  be  divided  into  two  groups.  The  first  group  consists  of 
controllable  factors  that  can  be  manipulated  or  decided  upon  in  advance  or 
during  the  mission.  The  second  group  consists  of  uncontrollable  or  noise  factors 
that  cannot  be  controlled  by  any  means.  For  instance,  while  choosing  the  speed 
to  cross  the  strait  is  a  controllable  factor,  the  number  of  enemy  forces  is  an 
uncontrollable  factor. 

In  total,  35  factors  are  varied  in  the  simulations,  and  they  are  discussed  in 
the  following  section.  Among  these  35  factors,  29  are  controllable  factors  and  6 
are  uncontrollable  factors.  The  NOB  Mixed  Design  spreadsheet  is  used  to 
generate  the  experimental  design  in  this  analysis  (Figure  13).  This  spreadsheet 
allows  studying  up  to  300  factors  with  512  design  points  (Vieira  Jr.,  Sanchez, 
Kienitz,  &  Belderrain,  2012). 
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Figure  13.  Partial  Screenshot  of  Nearly  Orthogonal  Nearly  Balanced  Mixed 

Design  spreadsheet. 


29 


The  NOB  Mixed  Design  spreadsheet  ensures  that  the  design  has  good 
space-filling  properties  and  that  the  factors  are  nearly  orthogonal.  Orthogonality 
has  some  advantages  when  fitting  a  meta-model,  because  when  the  factors  are 
uncorrelated,  both  the  computation  and  the  interpretation  of  the  response  surface 
simplifies  (Kleijnen  et  al.,  2005).  The  maximum  correlation  between  the  factors 
studied  in  this  thesis  is  less  than  0.03  which  is  considered  to  be  very  low.  A 
scatter  plot  of  a  portion  of  the  factors  and  the  mean  response  is  shown  in 
Figure  14.  The  space-filling  property  can  be  seen  with  this  graph. 
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Figure  1 4.  Scatter  plot  of  part  of  the  variables.  Space-filling  property  of  the  NOB 

design  can  be  seen  with  this  plot. 
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B.  CONTROLLABLE  FACTORS 


Controllable  factors  in  the  ASUW  model  consist  of  the  tactics,  ROE,  and 
physical  capabilities  of  HVU,  OPV,  and  the  helicopter.  There  are  three  ways  for 
the  decision  maker  or  naval  officer  to  control  these  factors: 

•  Selecting  the  capabilities  of  the  systems  that  form  the  naval 
combatant  prior  to  the  shipbuilding  process  (e.g.,  maximum  speed, 
presence  of  the  helicopter  platform).  Factors  of  this  type  cannot  be 
changed  once  the  ship  is  commissioned. 

•  Changing  the  properties  of  the  ship  following  its  commissioning 
(e.g.,  number  of  machine  guns,  radar  type/range).  Factors  of  this 
type  can  be  changed  after  the  commissioning;  however,  it  is  not 
likely  since  changing  these  factors  needs  a  significant  amount  of 
effort,  money,  and  man-hours. 

•  Deciding  how  to  use  the  current  capabilities  of  the  ship  in  a  naval 
operation  in  the  most  efficient  way  (e.g.,  tactics,  ROE). 

The  ASNET  PRONTO  NICOP  project  primarily  focuses  on  the  first  option, 
which  is  selecting  the  capabilities  of  the  surface  combatants  in  advance  while 
taking  into  consideration  the  operational  effectiveness  and  cost  effectiveness.  On 
the  other  hand,  the  author  believes  that  the  tactical/operational  decisions  or  ROE 
may  have  a  paramount  effect  on  the  outcome  of  a  mission,  and  the  concept  of 
operations  can  be  analyzed  along  with  the  capability  analysis.  Therefore,  the 
implications  of  various  tactical/operational  decisions  and  courses  of  action  (COA) 
should  be  considered  in  the  operational  effectiveness  studies.  For  example,  the 
maximum  speed  of  a  surface  combatant  is  a  capability  related  issue.  However, 
even  if  the  speed  of  the  ship  is  not  very  fast,  the  naval  officer  in  charge  may  be 
able  to  exploit  the  enemy’s  vulnerabilities  to  achieve  a  certain  goal  with  effective 
use  of  tactics.  For  this  reason,  not  only  capability  related  factors,  but  also  tactical 
and  ROE  related  factors,  are  studied  in  this  thesis. 
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1. 


HVU  Factors 


There  are  no  factors  that  are  directly  linked  to  the  capabilities  of  the  HVU; 
nevertheless,  some  factors,  such  as  speed  to  cross  the  strait,  indirectly  affect  the 
HVU  performance  in  the  mission.  These  factors  are  explained  in  “Tactics  and 
Rules  of  Engagement,”  later  in  this  chapter. 

2.  OPV  Factors 

a.  OPV  Speed 

In  many  scenarios  previously  implemented  in  MANA,  once  the 
speed  of  an  agent  was  set,  the  agent  could  only  move  in  that  specified  speed 
throughout  the  simulation  runs.  In  this  scenario,  on  the  other  hand,  the  OPV’s 
speed  is  defined  in  two  ways. 

•  When  there  is  no  enemy,  the  OPV  moves  at  the  speed  with 
which  the  HVU  passes  through  the  Strait  of  Gibraltar. 
Depending  on  the  design  point,  this  speed  can  be  as  low  as 
10  knots  and  as  high  as  20  knots.  The  speed  to  cross  the 
strait  is  related  to  tactics,  ROE,  and  regulations  at  sea.  It  is 
explained  in  “Tactics  and  Rules  of  Engagement,”  later  in  this 
chapter. 

•  When  a  friendly  asset  classifies  a  boat  as  hostile,  the  OPV 
moves  towards  the  hostile  boat  at  its  maximum  speed,  which 
ranges  from  20  knots  to  40  knots. 

b.  OPV  Weapons 

The  OPV  can  have  three  types  of  weapons  in  the  scenario:  a 
large/medium  caliber  main  gun,  a  small  caliber  auxiliary  gun,  and  a  machine  gun. 
Each  gun  type  has  a  different  set  of  values  for  the  following  parameters:  hit 
probability,  maximum  effective  range,  minimum  effective  range,  and  bursts  of  fire 
per  minute. 
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The  hit  probability  for  a  certain  weapon  is  classified  information  and 
cannot  be  obtained  from  an  open  source.  Moreover,  the  standard  deviation  of  a 
gun’s  probability  of  hit  is  relatively  high  when  compared  to  a  missile’s  probability 
of  hit.  The  reason  for  high  variation  is  that  the  guns  are  less  automated,  and 
there  are  more  factors  affecting  the  gun  fire.  For  instance,  it  is  very  difficult  to  hit 
a  small  speedboat,  but  it  is  relatively  easy  to  hit  a  tanker.  Therefore,  there  cannot 
be  a  fixed  hit  probability  that  fits  every  situation.  For  all  these  reasons,  the  hit 
probability  is  varied  in  the  analysis. 


Although  it  is  fairly  easy  to  set  most  of  the  weapon  properties  in 
MANA,  it  is  not  straightforward  to  set  the  inter-firing  time.  MANA  uses  the 
following  equation  for  calculating  the  shots  per  second: 


X 

— —  =  ShotsPerSecond 

100 


In  his  equation,  x  is  the  user  input  value.  For  instance,  if  the  user 
sets  x  to  100,  then  the  agent  can  fire  one  shot  per  second.  Unfortunately,  MANA 
does  not  provide  a  differentiation  between  a  single  shot  and  a  burst  of  fire.  While 
a  single  shot  makes  more  sense  for  missiles,  it  does  not  make  much  sense  when 
the  weapon  is  a  gun.  Since  the  weapons  used  in  this  scenario  are  guns,  a  burst 
of  fire  is  used  instead  of  a  single  shot  in  the  analysis.  The  factor  used  in  place  of 
“shots  per  second”  is  “bursts  per  minute.”  Therefore,  the  equation  below  is  used 
to  calculate  the  bursts  per  minute: 


60  ■  x 
100 


BurstsPerM inute 


A  burst  depends  on  several  factors  such  as  aiming  time,  distance, 
reloading,  target  acquisition,  and  user  preference  (Abel,  2009).  Thus,  instead  of 
using  a  fixed  value,  a  range  of  values  is  tested  in  the  experiment  to  see  the  effect 
of  the  inter-firing  time  to  the  response  surface. 

(1)  Main  Gun  (Weapon  1).  A  warship’s  main  gun  can  be 
a  large  caliber  gun  or  a  medium  caliber  gun.  Many  navies  prefer  medium  caliber 
guns  for  their  OPVs;  therefore,  the  ranges  for  the  main  gun’s  general 
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characteristics  were  chosen  based  on  a  medium  caliber  76  mm  naval 
gun  (Table  1). 


Factor 

Min 

Max 

Units 

Hit  Probability 

0.1 

0.9 

- 

Maximum  Effective  Range 

7000 

10000 

meters 

Minimum  Effective  Range 

1000 

2000 

meters 

Bursts  per  Minute 

1 

10 

bursts 

Table  1 .  OPV  main  gun  factors  and  ranges. 


The  main  gun’s  maximum  effective  range  is  substantially 
higher  than  the  auxiliary  gun’s  and  machine  gun’s  maximum  effective  ranges. 
Firing  from  a  long  range  is  particularly  important  in  conventional  warfare,  but  not 
necessarily  when  fighting  with  terrorists.  In  areas  like  the  Strait  of  Gibraltar,  there 
are  so  many  merchant  vessels  that  it  is  almost  impossible  to  classify  a  ship  within 
long  distances.  In  this  scenario,  the  only  way  to  classify  an  unknown  vessel  from 
a  long  distance  is  with  a  helicopter.  Therefore,  even  though  the  maximum 
effective  range  of  the  main  gun  ranges  from  7000  meters  to  10000  meters,  the 
OPV  cannot  use  its  main  gun  if  the  enemy  boat  is  not  classified  as  hostile. 

To  see  the  effect  of  the  main  gun  to  the  response  surface,  a 
factor  that  enables  or  disables  the  main  gun  is  also  added  to  the  experimental 
design.  This  may  help  the  decision  maker  to  better  understand  the  diverse 
effects  of  not  using  a  main  gun  in  such  missions. 

(2)  Auxiliary  Gun  (Weapon  2).  The  auxiliary  gun  for  the 
OPV  is  a  small  caliber  gun.  The  ranges  for  the  parameters  of  the  auxiliary  gun 
are  chosen  based  on  a  30  mm  naval  gun  (Table  2).  The  auxiliary  gun’s  presence 
is  important  especially  when  the  OPV  is  not  able  use  its  main  gun  for  some 
reason.  If  the  hit  probability  of  the  auxiliary  gun  is  high,  it  can  be  a  game  changer 
for  friendly  side. 
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Factor 

Min 

Max 

Units 

Hit  Probability 

0.1 

0.9 

- 

Maximum  Effective  Range 

1000 

3000 

meters 

Minimum  Effective  Range 

200 

400 

meters 

Bursts  per  Minute 

1 

10 

bursts 

Table  2.  OPV  auxiliary  gun  factors  and  ranges. 


(3)  Machine  Guns  (Weapons  3,  4,  5).  A  machine  gun  is 
operated  by  OPV  personnel,  and  it  has  a  relatively  short  effective  range  when 
compared  to  the  effective  ranges  of  the  main  and  the  auxiliary  guns.  Its  main 
purpose  is  to  warn  other  ships  and  to  protect  its  own  ship  from  small  targets. 

The  machine  guns  are  very  useful  in  crowded  areas,  such  as 
straits,  since  it  is  extremely  hard  to  classify  a  small  boat  from  a  long  distance.  It  is 
also  impossible  to  use  missiles  or  long  range  guns  at  shorter  distances. 
Additionally,  it  might  be  the  case  that  a  country’s  ROE  may  not  allow  their  naval 
vessels  to  start  firing  at  another  vessel  unless  it  approaches  within  a  certain 
distance,  such  as  500  meters.  In  this  case,  the  naval  vessel  can  use  its  machine 
guns  both  for  warning  the  approaching  vessel  and  for  protecting  itself. 

A  12.7  mm  machine  gun  is  used  as  a  base  for  the  ranges  of 
the  OPV’s  machine  guns  (Table  3).  The  minimum  effective  range  for  the  machine 
guns  is  fixed  at  zero. 


Factor 

Min 

Max 

Units 

Hit  Probability 

0.1 

0.9 

- 

Maximum  Effective  Range 

100 

150 

meters 

Bursts  per  Minute 

1 

10 

bursts 

Table  3.  OPV  machine  gun  factors  and  ranges. 


Another  set  of  factors  related  to  the  machine  guns  is  the 
number  of  machine  guns  used.  In  the  scenario,  the  OPV  can  have  at  most  three 
machine  guns.  Three  factors  are  dedicated  to  enabling  or  disabling  each 
machine  gun  in  the  experimental  design.  The  guns  are  all  interchangeable,  so  an 
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alternative  design  would  be  use  a  single  factor  to  set  the  total  number  of  guns. 
Our  approach  can  be  easily  extended  for  future  work,  if  differences  in  gunner 
capabilities  or  positioning  are  explored. 

3.  Helicopter  Factors 

a.  Helicopter  Presence 

Whether  or  not  a  helicopter  is  tasked  in  the  escort  mission  is  a 
factor  in  the  experimental  design.  Although  this  factor  can  be  related  to  the 
tactics,  it  may  also  be  related  to  the  design  of  the  ship.  If  the  ship  has  a 
helicopter  platform,  then  it  can  perform  helicopter  tasks.  If  not,  then  the  friendly 
assets  will  need  a  land-based  helicopter  tasked  to  support  their  mission. 

b.  Helicopter  Speed  and  Sensors 

The  high  speed  capability  of  the  helicopter  makes  it  one  of  the  most 
valuable  assets  of  a  warship.  It  can  perform  search,  detection,  and 
reconnaissance  operations  in  relatively  short  amounts  of  time,  with  high 
accuracy.  Technological  advances  also  allow  the  helicopters  to  use  cameras  that 
help  them  better  classify  the  targets.  The  factors  of  helicopters  that  are  related  to 
the  speed  and  the  sensors  are  shown  in  Table  4. 


Factor 

Min 

Max 

Units 

Max  Speed 

50 

180 

knots 

Detection  Max  Range 

5000 

15000 

meters 

Classification  Max  Range 

500 

2000 

meters 

Table  4.  Helicopter’s  maximum  speed  and  sensor  factors  and  ranges. 


Helicopter  speed  is  also  defined  in  two  ways. 

•  When  there  is  no  unknown  vessel,  the  helicopter  moves  at 
the  speed  with  which  the  HVU  passes  through  the  Strait  of 
Gibraltar. 
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•  When  the  helicopter  detects  an  unknown  vessel,  it  moves 
towards  that  target  for  classification  at  its  maximum  speed, 
which  ranges  from  50  knots  to  180  knots. 

The  friendly  assets  have  AIS  devices,  which  allow  them  to  classify 
almost  all  of  the  vessels  in  the  strait.  However,  there  are  still  some  vessels  that 
cannot  be  classified  via  AIS.  Once  those  ships  are  detected,  then  the  helicopter 
approaches  them  for  classification.  In  the  real  world,  the  classification  distance 
may  depend  on  weather  conditions,  capability  of  the  camera,  or  the  training  of 
the  operators.  Therefore,  the  classification  range  is  designed  to  be  factor  in  the 
experiment.  This  factor  ranges  from  500  meters  to  2000  meters. 

c.  Helicopter’s  Weapon 

A  12.7  mm  machine  gun  is  used  as  a  base  for  the  ranges  of  the 
helicopter’s  machine  gun  (Table  5).  The  minimum  effective  range  for  the  machine 
guns  is  fixed  at  zero. 


Factor 

Min 

Max 

Units 

Hit  Probability 

0.1 

0.9 

- 

Maximum  Effective  Range 

100 

150 

meters 

Bursts  per  Minute 

1 

10 

bursts 

Table  5.  Helicopter  machine  gun  factors  and  ranges. 


Whether  the  helicopter  has  a  weapon  or  not  is  also  a  factor.  In 
almost  half  of  the  simulation  runs,  the  helicopter  has  a  machine  gun  that  is 
operated  by  helicopter  crew.  In  the  other  half,  it  does  not  have  any  weapons.  In 
this  case,  the  helicopter’s  only  purpose  is  to  detect  enemy  targets  and  send  this 
information  to  the  OPV  and  HVU. 

4.  Tactics  and  Rules  of  Engagement 

From  a  military  point  of  view,  tactics  can  be  defined  as  the  art  of  using  the 
capabilities  of  the  naval  forces  in  a  battle,  whereas  ROE  are  directives  issued  by 
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governments  that  allow  or  limit  naval  assets  to  use  their  forces  ( DoD  Dictionary, 
2006).  Factors  related  to  tactics  and  ROE  are  shown  in  Table  6. 


Factor 

Min 

Max 

Units 

Speed  to  Cross  the  Strait 

10 

20 

knots 

Range  to  Start  Counter  Measures 

300 

6000 

meters 

Distance  Between  HVU  and  OPV 

500 

4000 

meters 

Distance  Between  HVU  and  Helicopter 

3000 

10000 

meters 

Leading  Ship  Choice 

OPV 

HVU 

- 

HVU’s  Response  to  Attack 

Evade 

Not  Evade 

- 

Table  6.  Factors  of  tactics  and  ROE. 


These  factors  are  mostly  dependent  on  the  decisions  of  the  officer  in 
tactical  command  (OTC).  There  is  no  right  choice  of  parameters  that  can  be  used 
in  every  task;  nevertheless,  the  results  of  this  analysis  may  yield  important 
insights  about  the  tactical  decisions  in  naval  escort  missions  and  some  areas  of 
ASUW.  These  factors  are  also  important  to  demonstrate  the  fact  that  choice  of 
different  tactics  or  different  COAs  may  result  in  different  set  of  warship 
characteristics. 

a.  Speed  to  Transit  the  Strait 

In  most  of  the  straits  and  canals  all  over  the  world,  the  coastal 
states  adopt  laws  and  regulations  to  prevent  the  collision  of  ships.  They  usually 
put  a  maximum  and  a  minimum  speed  limit  within  the  strait  or  canal.  However, 
depending  on  the  number  of  ships  and  the  situation,  the  ships  are  allowed  to  use 
higher  speeds. 

In  the  Strait  of  Gibraltar,  the  recommended  maximum  speed  limit  is 
13  knots  (Arceredillo,  Sagarminaga,  de  Stephanis,  Canadas,  &  Lago,  2008). 
Nevertheless  the  average  speed  is  higher  than  13  knots.  If  there  is  intelligence 
regarding  a  terrorist  boat  attack  in  the  strait,  with  which  speed  should  the  OTC 
order  the  convoy  to  cross  the  strait?  The  factor  “speed  to  cross  the  strait”  is 
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ranged  from  10  knots  to  20  knots  to  answer  this  question  and  to  understand 
whether  the  speed  has  an  effect  on  the  mission  success. 

b.  Range  to  Start  Counter  Measures 

Countries  have  ROE  that  allow  their  warships  to  start  firing  on  other 
ships  when  certain  conditions  are  met.  The  factor  “range  to  start  counter 
measures”  represents  the  distance  at  which  the  OPV  is  able  to  start  firing  if  the 
hostile  boat  is  within  the  maximum  effective  range  of  its  weapons.  This  assumes 
the  target  ship  is  classified  as  hostile  and  all  the  necessary  warnings  are  given. 
The  range  to  start  counter  measures  ranges  from  300  meters  to  6000  meters. 

c.  Distance  between  HVU  and  OPV 

The  distance  between  the  HVU  and  the  OPV  is  a  tactical  decision 
and  is  usually  made  by  the  OTC.  For  instance,  when  the  direction  of  the  enemy 
is  unknown,  it  may  be  a  good  idea  to  keep  the  distance  between  the  HVU  and 
OPV  small.  On  the  other  hand,  if  the  enemy  is  expected  from  a  certain  direction, 
it  may  be  a  better  idea  to  use  the  OPV  in  that  direction  with  a  greater  distance 
between  HVU  and  OPV.  However,  the  direction  from  which  the  hostile  boats 
approach  is  not  known  in  this  scenario,  and  this  is  usually  the  case  in  actual 
missions.  To  understand  how  the  distance  between  the  escort  ship  and  the  HVU 
affects  mission  success,  a  factor  is  added  to  the  experimental  design.  This  factor 
ranges  from  500  meters  to  4000  meters. 

d.  Distance  between  HVU  and  Helicopter 

The  distance  between  the  HVU  and  the  helicopter  is  also  a  tactical 
decision;  however,  it  may  not  be  as  important  as  the  distance  between  HVU  and 
OPV  due  to  the  helicopter’s  high  speed  capability.  This  factor  ranges  from  3000 
meters  to  10000  meters. 
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e.  Leading  Ship  Choice 

The  decision  regarding  the  formation  of  the  ships  is  a  tactical 
decision  as  well.  This  decision  is  particularly  important  when  the  convoy  does  not 
have  a  helicopter.  This  factor  has  two  options.  The  first  is  keeping  the  OPV  in  the 
forward  position  and  making  it  the  leading  ship.  The  second  is  making  the  HVU 
the  leading  ship. 

f.  HVU’s  Response  to  Attack 

The  HVU  has  two  options  when  the  friendly  assets  classify  a  ship 
as  hostile.  It  can  either  continue  on  the  convoy’s  waypoint  without  changing  its 
route,  or  it  can  change  its  route  to  evade  the  terrorist  boat. 

C.  UNCONTROLLABLE  FACTORS  (NOISE  FACTORS) 

1.  Hostile  Boat  Factors 

There  are  three  factors  regarding  hostile  boats:  the  number  of  boats,  the 
maximum  speed  of  the  boats,  and  whether  or  not  they  employ  tactics  to  evade 
the  OPV. 


a.  Number  of  Hostile  Boats 

The  number  of  terrorist  boats  is  a  factor  that  ranges  from  one  boat 
to  ten  boats.  The  location  of  the  boats  is  selected  randomly,  and  the  terrorists  are 
assumed  to  be  capable  of  communicating  with  each  other.  Depending  on  the 
location  of  each  individual  boat,  the  terrorist  boats  are  modeled  to  perform  a 
swarm  attack  on  the  HVU. 

b.  Hostile  Boat  Speed 

The  speed  of  the  terrorist  boats  ranges  from  20  knots  to  40  knots. 
The  speed  is  very  important  for  the  terrorist  boats  when  the  speed  of  the  OPV  is 
lower  than  their  speed,  and  when  the  number  of  bursts  per  minute  for  the  OPV’s 
weapon  is  not  high. 


40 


c.  Evade  OPV 

The  terrorists  can  employ  different  tactics  as  well.  The  scenario 
was  modeled  such  that  the  terrorist  boats  do  not  directly  attack  the  OPV.  Their 
only  target  is  the  HVU;  however,  they  are  aware  of  the  OPV.  Therefore,  it  may  be 
a  wise  choice  if  they  evade  the  warship,  while  at  the  same  time  approaching  the 
HVU.  Whether  the  terrorist  boats  evade  or  not  is  a  two-level  factor  that 
represents  the  enemy  tactics. 

2.  Known/Unknown  Vessel  Factors 

The  only  factor  regarding  the  known  and  unknown  neutral  vessels  is  their 
number.  The  initial  location  of  the  vessels  has  a  high  degree  of  randomness; 
however,  this  is  not  designed  to  be  factor.  The  ranges  of  the  neutral  ships’ 
factors  are  shown  in  Table  7. 


Factor 

Min 

Max 

Number  of  Known  Vessels 

10 

30 

Number  of  Unknown  Vessels 

1 

10 

Table  7.  Factors  of  known/unknown  vessels. 


D.  DATA  ANALYSIS 

1.  Simulation  Runs 

The  combat  model  for  the  operational  ASUW  scenario  was  built  using 
MANA  version  5.01.04.  The  author  started  the  analysis  with  100  replications  for 
each  of  the  512  design  points  of  the  DoE.  In  the  meantime,  New  Zealand’s  DTA 
fixed  several  errors  in  MANA,  which  were  mostly  GUI  related  errors,  and  in 
released  version  5.01.05.  This  version  was  not  tested  by  any  NPS  faculty  or 
students  before,  and  it  did  not  prove  to  be  operating  well.  Therefore,  both  for 
testing  the  new  version  of  MANA  and  testing  the  baseline  model,  another  100 
replications  were  run  using  the  new  version.  This  time,  a  different  set  of  columns 
for  factors  was  used  in  the  NOB  Mixed  Design  spreadsheet  to  increase  the 
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space-filling  property  of  the  design.  Both  sets  of  runs  were  analyzed,  and  the 
results  showed  that  the  two  versions  did  not  differ  for  this  particular  scenario. 
Thus,  the  two  concatenated  sets  of  runs,  which  sum  up  to  102400  runs  with  1024 
design  points,  are  used  throughout  the  data  analysis. 

In  the  initial  design,  the  time  step  of  the  model  was  also  a  factor.  The 
choice  of  time  step  is  important  because  it  directly  affects  the  total  time  required 
to  make  the  simulation  runs,  but  overly  large  time  steps  can  lead  to  unusual 
behavior  and  model  artifacts  (Buss  &  Al  Rowaei,  2010).  The  values  one,  two,  and 
five  were  tested  in  the  simulation  runs.  The  results  of  the  runs  with  time  step 
values  one  and  two  did  not  seem  to  be  different,  while  the  results  of  the  runs  in 
which  the  time  step  is  five  proved  to  be  slightly  different.  This  indicates  that  time 
steps  of  five  or  larger  should  be  avoided,  but  there  does  not  appear  to  be  a  need 
to  reduce  the  time  step  even  further.  For  this  reason,  and  for  the  fact  that  it  took 
less  than  24  hours  in  a  cluster  to  run  100  replications  of  this  scenario,  we 
decided  to  fix  the  time  step  to  one. 

2.  Regression  Analysis 

There  are  several  approaches  to  regression  analysis.  While  some 
techniques  yield  similar  results,  others  may  result  in  different  meta-models.  Two 
techniques  are  used  in  this  analysis:  logistic  regression  and  least  squares 
regression.  Additionally,  partition  trees  are  used  to  compare  the  significant 
factors  with  the  significant  factors  of  logistic  regression  and  least  squares 
regression  meta-models. 

a.  Logistic  Regression 

The  only  MOE  in  this  analysis  is  mission  success.  Therefore,  the 
result  of  each  scenario  run  is  either  a  success  or  a  failure.  This  produces  a  binary 
result,  and  fitting  a  logistic  regression  is  a  proper  way  to  deal  with  binary 
response.  The  distribution  of  the  response  in  the  raw  data  is  shown  in  Figure  15. 
Among  102400  replications,  friendly  assets  achieved  mission  success  in  74066 

of  the  runs.  The  probability  of  the  success  in  overall  replications  is  around  0.72. 

42 


▼  Response  (Mission  Success) 


a  Frequencies 


Level 

Count 

Prob 

0 

28334 

0.27670 

1 

74066 

0.72330 

Total 

102400 

1.00000 

N  Missing 

0 

2  Levels 


Figure  15.  Distribution  of  the  response. 


The  summary  of  the  logistic  regression  model  with  all  the  main 
effects  is  shown  in  Figure  16.  The  p-value  of  this  model  is  less  than  0.0001, 
which  proves  that  the  saturated  logistic  regression  model  is  significant  when 
compared  to  the  null  model. 


Whole  Model  Test 

Model 

-Log  Likelihood 

DF 

Chi  Square 

Prob>ChiSq 

Difference 

23004.326 

35 

46008.65 

<0001* 

Full 

37392.092 

Reduced 

60396.418 

Figure  1 6.  Summary  of  the  logistic  regression  model. 


The  leading  15  factors  of  the  logistic  regression  model  sorted  in  the 
order  of  significance  from  high  to  low  are  shown  in  Figure  17.  The  first  five 
factors  in  the  list  seem  to  have  a  huge  impact  on  the  response  surface,  whereas 
the  other  factors  have  much  less  impact.  Moreover,  whether  or  not  the  OPV  main 
gun  is  enabled  appears  to  be  the  most  significant  factor  of  all. 
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a  Parameter  Estimates 


Term 

Estimate 

Std  Error 

ChISquare 

Prot»ChlSq 

OPV_Weapon1_Enable|0| 

1  01602893 

0  0102704 

9786.7 

<.0001* 

RED_No  of  Agents 

0.33265595 

0003571 

86776 

<  0001* 

OPV_Forward  Pos!tlon|0] 

087987545 

0010339 

72424 

<0001* 

Range  to  start  counter  measures 

*0.0004297 

5  8753e-6 

5349  3 

<  0001* 

OPV_Weapon2_Probablllty 

-2.6949762 

0  0380387 

5019.5 

<0001* 

Helo_Adlve|0] 

0.55865071 

0  0096813 

3329.8 

<.0001* 

RED_Movement  Speed 

0.06888563 

0  0016536 

1735.3 

<0001* 

OPV_Weapon1_Probab!llty 

-1  4829545 

0  0360533 

1691  9 

<  0001* 

HVU_Evade|0] 

0  37153035 

0  0094871 

1533.6 

A 

o 

o 

o 

• 

OPV_Weapon2_ShotperSecond 

-0.0650704 

0  0019035 

11686 

<.0001* 

OPV_Max  Speed 

-0046021 

0  0016267 

80042 

<  0001* 

OPV_Weapon1_ShotperSecond 

-0.0434965 

0  0018913 

52894 

<.0001* 

OPV_HVU_Distance 

0  00018018 

9.2733e-6 

377  52 

<.0001* 

RED_Evold  OPV  (with  levels) 

001189559 

0  0006433 

341  98 

A 

o 

o 

o 

— k 

OPV_Weapon4_Enable(0) 

0.17035682 

0  0092978 

335.70 

<.0001* 

Figure  1 7.  Parameter  estimates  for  the  logistic  regression. 


The  purpose  of  this  analysis  is  not  predicting  the  mission  success 
of  an  OPV  in  an  ASUW  mission,  but  rather  understanding  the  factors  affecting 
the  mission  success.  Nevertheless,  it  is  still  possible  to  assess  how  well  the 
meta-model  predicts  the  mission  success.  The  Receiver  Operator  Characteristic 
(ROC)  curve  is  one  way  to  evaluate  the  meta-models.  As  a  rule  of  thumb,  when 
the  area  under  the  ROC  curve  is  between  0.8  and  0.9,  the  meta-model  is 
considered  to  be  predicting  well;  when  it  is  higher  than  0.9,  the  meta-model  is 
considered  to  be  predicting  excellently  (Hemmingsson,  Udden,  &  Neovius,  2009). 
The  ROC  curve  of  the  logistic  regression  model  is  demonstrated  in  Figure  18. 
The  area  under  the  curve  is  0.89,  which  proves  that  the  meta-model  is  predicting 
fairly  well. 
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Figure  1 8.  The  ROC  curve  for  logistic  regression  model  evaluation. 

b.  Least  Squares  Regression 

For  any  specific  design  point,  even  though  all  the  values  of  the 
input  parameters  are  the  same,  each  simulation  run  (called  a  replication) 
produces  different  results  due  to  the  randomness  embedded  in  the  modeling 
platform.  A  designed  experiment  gives  the  analyst  an  opportunity  to  investigate  a 
range  of  outcomes  based  on  different  parameter  settings,  but  it  is  sometimes 
better  to  simplify  the  data  to  gain  more  insights.  For  instance,  when  dealing  with 
a  binary  response,  it  is  possible  to  take  the  mean  of  the  response  for  all  the 
replications  at  each  design  point.  Then  the  aggregated  average  response  values 
can  be  used  to  predict  mission  success.  Taking  the  mean  of  the  response  for  all 
replications  of  each  design  point  yields  a  multi-level  response  ranging  from  zero 
to  one.  In  this  way,  the  analyst  can  fit  a  regular  regression  model  to  the 
aggregated  data,  and  this  might  provide  a  better  insight  into  the  problem  at  hand 
because  it  is  easy  to  interpret. 

The  distribution  of  the  mean  of  the  response  in  the  aggregated  data 
is  shown  in  Figure  19.  The  overall  proportion  of  mission  success  is  0.7233,  which 
is  equal  to  the  proportion  of  mission  success  in  the  raw  data.  It  is  also  worth 
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noting  that  there  is  a  large  number  of  instances  where  the  proportion  of  mission 
success  is  greater  than  0.95  or  lower  than  0.05.  This  means  it  will  be  more 
difficult  for  a  polynomial  model  to  do  an  effective  job  of  fitting  the  aggregated 
response  data.  Even  so,  it  is  interesting  to  see  what  factors  are  most  important. 

Stepwise  regression  methods  and  expert  judgment  are  used  to 
select  the  significant  factors  to  fit  the  least  squares  regression  model.  The 
threshold  for  the  p-values,  or  the  significance  level,  which  is  used  as  evidence  of 
statistical  significance  for  terms  included  in  the  models  (Htoon  &  Chan,  2010),  is 
set  to  be  very  low  to  avoid  over-fitting.  This  low  p-value  threshold  also  reduces 
the  chances  of  “false  positives”  (Type  I  errors)  when  identifying  important  factors 
and  allows  us  to  have  interpretable  results.  It  is  also  true  that  for  large  data  sets, 
such  as  those  resulting  from  large-scale  simulation  experiments,  regression 
models  can  contain  terms  that  are  statistically  significant  but  not  practically 
important.  A  low  p-value  threshold  is  useful  for  obtaining  relatively  parsimonious 
models  in  these  situations. 
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21  Summary  Statistics 

Mean  0.7233008 

Std  Dev  0.3556502 

Std  Err  Mean  0.0111141 
Upper95%Mean  0.7451098 
Lower  95%  Mean  0.7014918 
N  1024 


Figure  19.  Distribution  of  the  mean  response.  The  total  number  of  occurrences 
(N)  is  1024.  Among  those  occurrences,  187  of  them  resulted  in  with 
probability  of  mission  success  greater  than  0.95. 
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(1)  Main  Effects  Model.  The  Actual  by  Predicted  plot  and 
the  Summary  of  the  stepwise  model,  fitted  with  only  the  main  factors  without  any 
interactions,  are  shown  in  Figure  20.  Only  15  factors,  which  proved  to  be 
significant,  are  included  in  the  model.  The  R-Square  value  is  around  0.56.  This 
value  represents  how  much  variability  can  be  explained  using  this  model.  There 
is  considerable  randomness,  as  is  often  the  case  in  the  combat  models,  but  the 
regression  terms  can  help  the  analyst  focus  on  what  really  affects  the  mission 
success. 


Summary  of  Fit 

RSquare 

0.559607 

RSquare  Adj 

0.553054 

Root  Mean  Square  Error 

0.237766 

Mean  of  Response 
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Observations  (or  Sum  Wgts) 

1024 
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Figure  20.  Actual  by  predicted  plot  and  the  summary  of  the  fit  for  the  main  effects 

regression  model. 


The  sorted  parameter  estimates  of  the  main  effects  model  are 
shown  in  Figure  21.  Note  that  the  first  five  factors  have  a  more  significant  effect 
on  the  probability  of  mission  success  when  compared  to  others,  as  can  be  seen 
by  the  magnitudes  of  the  t  ratios.  These  five  factors  are  exactly  the  same  factors 
that  showed  up  in  the  logistic  regression  model,  as  expected. 
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Sorted  Parameter  Estimates 

Term 

Estimate 

OPV_Weapon1_Enable[0] 

-0.128137 

RED_No.of  .Agents 

-0.040321 

OPV_Forward  Positional 

-0.102261 

Range  to  start  counter  measures 

5.503e-5 

OP  V_We  a  p  o  n2_P  ro  b  a  b  i  lity 

0.3509198 

Helo_Active[Q| 

-0.062972 

RED_Movement  Speed 

-0.00801 

OPV_Weapon1_Probability 

0.176017 

HVU_Evade[0] 

-0.042083 

OP  V_We  a  p  o  n2_S  h  otp  e  rS  e  cond 

0.0084448 

OPV_Max  Speed 

0.0047018 

OPV_Weapon4_Enable[0] 

-0.022876 

RED_Evoid  OPV(with  levels) 

-0.001535 

OP  V_H  VU_D  i  sta  n  ce 

-2.09e-5 

OP  V_We  a  p  o  n  1  _S  h  otp  e  rS  e  cond 
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0.007443 

-8.46 
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0.001507 
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Figure  21 .  Sorted  parameter  estimates  of  the  main  effects  regression  model. 


(2)  Second  Order  Model.  The  Actual  by  Predicted  plot 
and  the  Summary  of  Fit  built  with  the  main  effects,  two-way  interactions,  and 
polynomials  are  shown  in  Figure  22.  The  R-Squared  value  of  the  second  order 
model  is  higher  than  that  of  the  main  effects  model,  even  though  the  second 
order  model  contains  one  less  term  and  involves  fewer  factors.  One  might  think 
that  is  why  the  R-squared  value  is  not  very  high. 


A  Actual  by  Predicted  Plot 


A  Summary  of  Fit 

RSquare 

RSquare  Adj 

Root  Mean  Square  Error 

Mean  of  Response 

Observations  (or  Sum  Wgts) 


0.604107 

0.598614 

0.225322 

0.723301 

1024 


Mean(Response  (Mission  Success)) 
Predicted  P<  0001  RSq=0.60  RMSE=0.2253 


Figure  22.  Sorted  parameter  estimates  for  the  stepwise  regression  model  to 
include  the  main  effects,  two-way  interactions,  and  polynomials. 
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The  sorted  parameter  estimates  are  shown  in  Figure  23.  The 
six  most  significant  factors  are  the  same  as  the  factors  of  the  main  effects  model, 
with  a  slightly  different  order. 


Sorted  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

OPV  Weaponl  EnableM 

-0.122462 

0.00707 

-17.32 

<.0001* 

RED  No.of.Agents 

-0.038986 

0.002432 

-16.03 

1 

<0001* 

OPV  Forward  Position[0] 

-0.102047 

0.007061 

-14.45 

_ 

<0001* 

OPV  Weapon2  Probability 

0.3629253 

0.027438 

13.23 

1 

<0001* 

Range  to  start  counter  measures 

5.41 67e-5 

4.289e-6 

12.63 

r 

<.0001* 

Helo  Active[0] 

-0.063289 

0.007052 

-8.97 

:  ■ 

<.0001* 

(Range  to  start  counter  measures-3150)*(Range  to  start  counter  measures-3150) 

-2.251  e-8 

2.919e-9 

-7.71 

:1 

A 

o 

o 

o 

HVU_Evade[0] 

-0.044864 

0.007055 

-6.36 

A 

o 

CD 

CD 

RED  Movement  Speed 

-0.007591 

0.00122 

-6.22 

<.0001* 

OPV  Weaponl  Enable[0]*(OPV  Weapon2  Probability-0.48252) 

0.1618093 

0.027541 

5.88 

A 

o 

CD 

CD 

OPV  Weaponl  Enable[0]*OPV  Forward  Positions 

-0.041445 

0.00708 

-5.85 

1 

A 

O 

CD 

CD 

OPV_Weaponl_Probabilily 

0.1557493 

0.026901 

5.79 

J; 

<.0001* 

(Rangeto  startcountermeasures-3l50)*Helo_Active[0] 

0.0000234 

4.287e-6 

5.46 

J 

<.0001* 

OPV_Weapon2_SbotperSecond 

0.0073734 

0.001431 

5.15 

I:  : 

<.0001* 

Figure  23.  Sorted  parameter  estimates  of  the  second  order  model. 


Some  terms  of  the  second  order  model  are  polynomials  and 
two-way  interactions.  For  example,  range  to  start  counter  measures  is  a 
quadratic  term.  That  is,  keeping  all  other  factors  constant,  changing  the  value  of 
this  term  changes  the  value  of  the  mission  success  non-linearly. 

When  doing  a  regression  analysis,  it  is  also  possible  to  fit  the 
model  with  more  degrees  of  interactions  and  polynomials.  As  a  part  of  this 
regression  analysis,  three-way  interactions  were  tested  as  well;  however,  they 
did  not  prove  to  be  significant. 

3.  Partition  Tree 

An  important  technique  for  demonstrating  the  contribution  of  the  factors  to 
the  response  surface  is  building  a  partition  tree.  There  are  several  other  names 
given  to  this  type  of  regression  model,  including  Classification  and  Regression 
Tree  (CART),  and  decision  tree.  The  partition  tree  is  mainly  useful  because  it  is 
easy  to  understand  both  for  the  analyst  and  the  decision  maker.  It  can 
demonstrate  interactions,  is  relatively  easy  to  build,  and  allows  the  analyst  to 
easily  explore  the  relationship  between  factors  and  the  response.  Furthermore,  it 
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can  handle  both  numeric  and  categorical  data  (De’ath  &  Fabricius,  2000). 
Partition  trees  are  often  better  at  fitting  jumps  or  plateaus  in  the  data,  and  so  they 
provide  a  convenient  alternative  to  polynomial  regression  models  when  the 
response  is  not  necessarily  smooth. 

A  partition  tree  can  be  built  using  either  the  initial  data  with  binary 
response  or  the  aggregated  data  with  continuous  response.  Both  partition  trees 
produce  a  similar  tree  structure.  The  contribution  of  the  factors  to  the  partition 
tree  is  similar  as  well.  The  partition  trees  used  in  this  analysis  were  built  on  the 
raw  data,  which  consists  of  102400  data  points  and  binary  response. 

When  forming  the  partition  tree,  it  is  also  possible  to  choose  the  set  of 
factors  on  which  the  splits  may  occur.  A  full  set  of  factors  will  provide  insights  on 
the  relative  importance  of  all  the  factors,  whereas  a  small  set  of  factors  will  help 
the  analyst  to  see  the  effects  of  specific  factors  to  the  response  surface.  In  the 
first  part  of  partition  tree  analysis,  all  factors  are  included  in  the  tree  structure.  In 
the  second  part,  only  the  controllable  factors  are  used  to  build  the  tree. 

In  the  following  parts  of  the  partition  tree  analysis,  the  color  blue 
represents  the  mission  success  of  the  friendly  assets,  and  the  color  red 
represents  the  mission  failure  or  enemy  success. 

a.  Partition  Tree  to  Include  All  Factors 

In  this  part  of  the  analysis,  all  the  controllable  and  uncontrollable 
factors  are  included  in  the  set  of  factors  where  the  splits  can  occur.  A  portion  of 
the  partition  tree  for  the  mission  success  is  shown  in  Figure  24.  The  probability  of 
mission  success  or  mission  failure  is  demonstrated  in  the  rectangles  for  each  leaf 
of  the  tree.  For  example,  the  first  split  of  the  partition  tree  occurs  with  the  factor 
“range  to  start  counter  measures.”  When  the  range  to  start  counter  measures  is 
greater  or  equal  to  1282  meters,  the  probability  of  mission  success  is  0.7749. 
When  this  range  is  lower  than  1282  meters,  the  probability  of  mission  success  is 
0.4745. 
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Figure  24.  Portion  of  the  partition  tree  to  include  all  the  variables.  Blue  represents 
the  probability  of  mission  success  and  red  represents  the  probability  of 

mission  failure. 


In  this  partition  tree,  the  second  and  third  splits  occur  on  whether 
the  main  gun  is  available  or  not,  and  the  probability  of  hit  for  the  auxiliary  gun, 
respectively.  If  the  range  to  start  counter  measures  is  greater  than  or  equal  to 
1282  meters,  and  if  the  main  gun  is  enabled,  then  the  friendly  assets  have  almost 
a  90%  chance  of  mission  success. 

The  partition  of  the  data  can  also  be  depicted  with  a  partition  graph 
as  shown  in  Figure  25.  In  this  graph,  each  hierarchical  level  from  down  to  top 
represents  the  levels  of  the  partition  tree. 
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Figure  25.  Partition  graph  for  mission  success  to  include  all  the  factors.  Blue 
represents  the  probability  of  mission  success  and  red  represents  the 
probability  of  mission  failure. 


When  the  OPV  starts  firing  early  and  uses  the  main  gun,  the 
probability  of  mission  success  is  quite  high.  But  what  if  the  OTC  does  not  want  to 
use  the  main  gun,  or  what  if  the  main  gun  needs  maintenance?  In  this  case,  the 
probability  of  hit  for  the  auxiliary  gun  comes  into  play  as  an  important  factor.  If  the 
probability  of  hit  of  the  auxiliary  gun  is  greater  than  0.4,  and  the  OPV  is  in  the 
forward  position  as  a  leading  ship,  then  the  friendly  assets  can  still  achieve  high 
probability  of  mission  success.  On  the  other  hand,  if  the  hit  probability  of  the 
auxiliary  gun  is  lower  than  0.4  and  there  are  four  or  more  hostile  boats,  the 
probability  of  success  decreases  to  0.25. 

The  ROE  specify  how  close  the  hostile  boats  must  be  for  the  ship 
to  start  counter  measures.  If  this  distance  is  short,  then  having  a  helicopter 
becomes  more  important.  The  reason  for  that  is  the  OPV  does  not  leave  the  HVU 
alone  to  go  ahead  and  classify  the  unknown  targets.  When  there  is  no  helicopter, 
the  only  way  for  classification  is  by  visually  analyzing  the  target  and  warning  it.  If 
the  target  does  not  seem  to  care  about  the  warnings  and  continues  its  approach, 
then  it  is  classified  as  enemy  within  a  certain  distance  depending  on  the  ROE. 
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When  the  helicopter  is  available  and  it  detects  an  unknown  target,  it  goes  above 
the  target  to  classify  it.  In  this  case,  even  when  the  target  is  farther  away  from  the 
HVU,  it  can  be  classified  as  hostile  or  neutral;  the  friendly  assets  can  start 
counter  measures  if  it  is  classified  as  hostile. 

When  there  are  ten  splits  in  the  partition  tree,  the  contribution  of  the 
factors  to  the  mission  success  is  shown  in  Figure  26.  The  factors  that  are 
significant  are  similar  to  the  significant  factors  of  the  regression  models  as 
discussed  in  more  detail  later  in  this  chapter. 


a  Column  Contributions 

Number 


Term 

of  Splits 

GA2 

OPV_Weapon1_Enable 

1 

7060.9097 

RE  D_No.  of.  Agents 

4 

6820.0773 

Range  to  start  counter  measures 

1 

6002.9402 

OPV_Weapon2_Probability 

1 

3857.9185 

OPV_Forward  Position 

1 

3005.2892 

Helo_Active 

1 

2520.1978 

OPV_Max  Speed 

1 

1087.7391 

Figure  26.  Factor  contributions  for  partition  tree  with  1 0  splits,  both  controllable 

and  uncontrollable  factors. 


b.  Partition  Tree  to  Include  Controllable  Factors  Only 

In  a  naval  mission,  the  decision  maker  has  very  little  or  no  control 
of  the  factors  related  to  the  enemy  or  to  the  merchant  traffic.  Therefore,  what  the 
analyst  should  pay  more  attention  to  is  the  controllable  factors.  For  instance,  the 
decision  maker  may  not  be  able  to  change  the  number  of  hostile  boats,  but  he  or 
she  can  change  the  number  of  machine  guns  on  the  OPV.  The  partition  tree  with 
only  the  controllable  factors  included  in  the  model  is  demonstrated  in  Figure  27. 
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Figure  27.  Portion  of  the  partition  tree  to  include  controllable  factors  only.  Blue 
represents  the  probability  of  mission  success  and  red  represents  the 
probability  of  mission  failure. 


Since  most  of  the  significant  factors  are  controllable,  the  structure 
of  the  tree  does  not  change  significantly.  The  number  of  the  enemy  boats  does 
not  show  up  in  the  tree  since  it  is  an  uncontrollable  factor.  The  partition  graph 
shown  in  Figure  28  gives  a  more  detailed  split  of  the  data. 
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Figure  28.  Partition  for  mission  success  to  include  controllable  factors  only.  Blue 
represents  the  probability  of  mission  success  and  red  represents  the 
probability  of  mission  failure. 


At  the  top  level,  the  contribution  of  the  OPV’s  speed  to  the  mission 
success  seems  to  be  important  in  certain  situations.  It  has  been  discussed  that 
when  the  OPV  starts  counter  measures  from  a  long  distance  and  cannot  use  the 
main  gun,  then  the  hit  probability  of  the  auxiliary  gun  becomes  a  significant 
factor.  Yet,  if  the  OPV  is  not  the  leading  ship,  there  is  still  high  risk  in  mission 
success  when  the  OPV’s  maximum  speed  is  lower  than  25  knots. 

An  interesting  split  shown  in  Figure  28  is  the  effect  of  the  main 
gun’s  hit  probability  to  the  mission  success.  Although  having  a  higher  probability 
of  hit  increases  the  chance  of  mission  success,  it  does  not  improve  the  outcome 
substantially.  More  discussion  of  the  hit  probabilities  appears  later  in  this  chapter. 

E.  FACTOR  SIGNIFICANCE 

The  key  drivers  of  the  shipbuilding  process  are  the  operational 

effectiveness  and  the  cost  effectiveness.  Decision  makers  of  the  naval 

acquisition  and  shipbuilding  processes  want  to  minimize  the  cost  while,  at  the 
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same  time,  maximize  the  operational  effectiveness  of  the  naval  combatants.  On 
the  other  hand,  decision  makers  at  sea,  the  naval  officers,  do  not  primarily  care 
about  the  acquisition  cost.  Their  purpose  is  accomplishing  the  mission.  Factor 
significance  is  paramount  for  both  groups  for  understanding  the  relative 
importance  of  factors  in  the  shipbuilding  process  and  in  the  operational 
environment. 

For  the  ASUW  scenario  studied  in  this  thesis,  the  comparison  of  the  key 
factors  (either  controllable  or  uncontrollable)  and  their  rankings  in  the  meta¬ 
models  are  given  in  Table  8.  The  first  six  factors  showed  up  in  all  of  the 
regression  models,  and  they  proved  to  be  significant  with  slightly  different 
rankings. 


Factors 

Logistic 

Regression 

Model 

Main 

Effects 

Model 

Second 

Order 

Model 

Partition 

Tree 

Main  Gun  Presence 

1 

1 

1 

1 

Number  of  Hostile  Boats 

2 

2 

2 

2 

Leading  Ship  Choice 

3 

3 

3 

5 

Range  to  Start  Counter  Measures 

4 

4 

5 

3 

Auxiliary  Gun  Hit  Probability 

5 

5 

4 

4 

Helicopter  Presence 

6 

6 

6 

6 

OPV  Speed 

11 

11 

- 

7 

Table  8.  A  comparison  of  the  significant  factors  and  their  rankings  in  different 

regression  models  and  partition  trees. 


The  main  gun  is  the  most  significant  factor  affecting  the  outcome  of  the 
mission.  When  the  main  gun  is  enabled,  the  result  is  generally  in  the  friendly 
force’s  favor.  On  the  other  hand,  if  the  main  gun  is  not  enabled,  but  the 
probability  of  hit  for  the  auxiliary  gun  is  high,  the  friendly  assets  can  still  achieve 
favorable  results. 

The  second  most  important  factor  appears  to  be  the  number  of  hostile 
boats.  It  is  obvious  that  when  the  number  of  enemies  increases,  the  friendly  force 
is  less  likely  to  win.  This  is  an  uncontrollable  factor;  although  the  decision  makers 
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should  consider  the  number  of  the  enemy  in  their  plans,  there  is  almost  no  way  to 
control  this  in  the  operational  environment.  However,  there  are  two  alternatives. 
First,  it  is  possible  for  the  analyst  to  choose  a  more  likely  value  as  an  input  and  to 
perform  the  analysis  using  the  fixed  value.  For  instance,  if  the  expected  number 
of  terrorist  boats  is  at  most  two,  the  data  analysis  can  be  performed  taking  two  as 
constant  when  scoping  the  scenario.  Second,  if  a  robust  alternative  is  found,  that 
means  that  while  the  number  of  enemies  has  some  effect  on  the  overall  mission 
success,  the  decision  maker’s  plans  do  not  depend  on  advance  knowledge  of  the 
enemy  strength. 

Leading  ship  choice  is  another  significant  factor  that  showed  up  in  all  of 
the  regression  models.  This  factor  is  purely  dependent  on  tactical  decisions,  and 
the  author  believes  that  this  is  something  the  decision  makers  will  love  to  see.  An 
important  factor  that  is  not  directly  related  to  the  capabilities  of  the  ship  may 
eventually  result  in  substantial  reduction  of  the  cost  of  the  naval  ships.  If  a 
warship  can  achieve  the  same  goal  with  less  capability,  why  not  buy  the  cheaper 
ship  instead  of  the  expensive  one? 

Range  to  start  counter  measures  is  mainly  related  to  the  ROE,  but  also 
related  to  the  tactics  and  the  capabilities  of  the  ships.  Generally,  if  the  ROE 
permits,  starting  to  fire  early  increases  the  chance  of  mission  success.  If  not, 
then  other  factors  such  as  hit  probability  become  more  important. 

A  helicopter  is  one  of  the  key  assets  of  a  warship;  however,  it  is  not  as 
important  in  every  operational  scenario.  If  the  ship  is  tasked  in  a  search  and 
rescue  mission,  then  the  helicopter  will  possibly  be  the  most  significant  factor.  In 
this  scenario,  on  the  other  hand,  it  is  not  one  of  the  top  five  significant  factors. 

OPV  maximum  speed  does  not  appear  to  be  a  key  factor  in  the  ASUW 
scenario  studied  in  this  thesis.  If  the  maximum  speed  of  the  OPV  is  not  very  high, 
this  can  be  compensated  with  tactical  decisions  such  as  ordering  the  ship  to  be 
the  leading  ship. 
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F. 


DISCUSSION  AND  OPERATIONAL  INSIGHTS 


The  technological  advances  in  electronics  led  engineers  to  build  powerful 
warships  since  the  1900s.  Some  people  began  to  think  that  the  more  capable  the 
ship,  the  greater  the  chance  of  mission  success.  Although  this  idea  might  be  true 
in  some  cases,  the  capability  of  the  ships  might  not  be  the  key  driver  of  mission 
success  in  every  task. 

There  are  important  practical  implications  of  the  data  analysis  done  in  this 
thesis.  The  naval  officers  should  be  aware  of  the  fact  that  tactical  decisions  may 
be  a  game  changer  in  favor  of  the  side  which  uses  tactics  appropriately.  Several 
examples  of  practical  implications  and  operational  insights  gained  from  this  study 
are  explained  in  the  following  paragraphs. 

1.  Weapons 

Most  of  the  surface  combatants  throughout  the  world  have  main  guns. 
Nevertheless,  it  might  be  the  case  that  OPV  is  not  allowed  to  use  the  main  gun 
due  to  the  ROE,  due  to  malfunction,  or  because  there  are  merchant  vessels 
nearby.  In  such  situation,  the  OPV  needs  an  auxiliary  gun  with  high  probability  of 
hit.  In  many  cases,  however,  increasing  the  probability  of  hit  of  the  auxiliary  gun 
may  not  be  possible.  The  best  way  to  overcome  this  problem  appears  to  be  well 
trained  personnel.  This  will  ensure  that  the  hit  probability  of  the  machine  guns  will 
turn  this  disadvantage  into  an  advantage.  Thus,  the  commanding  officer  (CO)  of 
a  ship  should  pay  more  attention  to  the  training  of  the  machine  gun  operators, 
especially  when  deployed  in  areas  with  a  high  number  of  merchant  vessels 
present. 

Although  the  presence  of  the  main  gun  appears  to  be  one  of  the  most 
influential  factors,  its  probability  of  hit  did  not  show  up  as  important.  This  made 
the  author  question  the  model  and  the  analysis.  Further  investigation  of  the  data 
showed  that  there  are  sound  reasons  to  explain  why  the  main  gun’s  hit 
probability  did  not  show  up  as  significant.  A  graph  of  main  gun’s  hit  probability 
versus  the  probability  of  mission  success  is  shown  in  Figure  29.  Note  that  this 
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graph  only  shows  the  mission  success  in  the  cases  where  the  main  gun  is 
enabled.  The  probability  of  mission  success  increases  slightly  with  the  probability 
of  hit  up  until  the  probability  is  equal  to  0.5.  At  first  glance,  probability  of  hit  higher 
than  0.5  does  not  seem  to  make  an  additional  impact  on  the  outcome  of  the 
mission.  One  of  the  reasons  might  be  that  it  may  be  more  difficult  to  increase  the 
chance  of  mission  success  when  it  approaches  100%. 
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Figure  29.  Main  gun’s  hit  probability  vs.  mission  success  probability. 


A  Tukey-Kramer  test  performed  on  the  data  proved  that  the  probability  of 
mission  success  is  not  statistically  different  when  the  hit  probability  is  greater 
than  or  equal  to  0.3  (Figure  30).  The  practical  implication  is  this:  having  a  main 
gun  with  probability  of  hit  equal  to  0.3  will  give  similar  outcomes  when  compared 
to  a  main  gun  with  probability  of  hit  equal  to  0.9.  This  information  yields  important 
insight:  increasing  a  gun’s  probability  of  hit  may  not  always  increase  the  chance 
of  mission  success  significantly.  Flowever,  it  decreases  the  variation  and, 
therefore,  decreases  the  risk  of  catastrophic  failure.  Greater  probability  of  hit 
results  in  greater  robustness. 
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A  Connecting  Letters  Report 
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Figure  30.  Connecting  letters  report  for  the  Tukey-Kramer  test.  Levels  not 
connected  by  same  letter  are  significantly  different. 


Another  insight  gained  from  this  analysis  is  that  in  the  ship  acquisition 
process,  it  may  not  be  crucial  to  buy  the  “most  advanced”  technology.  The  naval 
combatant  might  be  able  to  accomplish  the  mission  with  less  capable  and 
cheaper  systems. 

2.  Helicopter 

A  challenging  problem  for  the  OTC  or  the  CO  is  to  decide  when  to  start 
firing  at  the  hostile  targets.  If  the  OPV  is  not  allowed  to  fire  from  a  long  distance 
because  of  the  ROE,  or  if  the  CO  is  willing  to  wait  to  be  sure  about  the 
classification  of  the  enemy,  then  the  importance  of  having  a  helicopter  in  the 
mission  becomes  more  obvious.  The  helicopter  ensures  that  the  friendly  assets 
classify  a  ship  as  hostile  from  a  longer  distance.  Once  it  is  classified,  then  there 
is  no  problem  firing  at  hostile  boats  at  long  distances  from  the  HVU. 

3.  Range  to  Start  Counter  Measures 

In  “Data  Analysis”  it  is  mentioned  that  the  range  to  start  counter  measures 
is  a  quadratic  term  of  the  second  order  model.  What  does  it  mean  in  terms  of 
naval  operations?  A  snapshot  of  the  prediction  profiler  for  this  factor  is  shown  in 
Figure  31.  Initially,  the  probability  of  mission  success  increases  rapidly  with  the 
range;  however,  when  the  range  approaches  to  3000s,  increasing  the  range 
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does  not  affect  the  mission  success  as  much.  Furthermore,  since  the  friendly 
force  does  not  have  an  unlimited  number  of  rounds  and  since  the  weapons  can 
malfunction  with  intense  use,  starting  to  fire  too  early  may  result  in  a  decrease  in 
the  probability  of  mission  success. 


Figure  31 .  Prediction  profiler  for  range  to  start  counter  measures. 

This  shows  that  analytical  tools  should  be  used  to  enhance  our  knowledge 
about  even  obvious  tactical  decisions  and  ROE.  Moreover,  the  author  believes 
that  many  of  ROE  may  be  tested  in  the  simulation  environments,  and 
reconsidered  in  the  light  of  technological  advances  and  experience  periodically. 

4.  Leading  Ship  Choice 

The  analysis  in  this  thesis  showed  how  a  tactical  decision  might  impact 
the  outcome  of  a  mission.  Leading  ship  choice  is  a  tactics  related  factor,  and  it 
helps  the  naval  officer  to  better  understand  the  nature  of  the  naval  warfare. 

When  the  convoy  is  navigating  eastward  as  in  this  scenario  and  the  OPV 
is  the  leading  ship,  one  of  these  three  events  might  happen: 

•  The  hostile  boats  may  approach  from  the  east  as  shown  in  Figure 
32.  In  this  case,  the  OPV  can  quickly  impede  them  since  it  is 
between  the  FIVU  and  the  hostile  boats. 
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Figure  32.  Hostile  boat  approaches  from  east. 


•  The  hostile  boat  may  approach  from  the  west  as  depicted  in  Figure 
33.  In  this  case,  since  the  relative  speed  between  the  convoy  and 
the  hostile  boat  is  smaller,  it  takes  much  more  time  for  the  hostile 
boats  to  reach  the  convoy.  When  the  OPV  changes  its  course  to 
the  west  and  navigates  towards  the  hostile  boat,  it  increases  the 
relative  speed  between  the  hostile  boat  and  itself,  and  again  quickly 
impedes  the  hostile  boat  while  the  HVU  continues  on  its  course. 


Figure  33.  Hostile  boat  approaches  from  the  west. 
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•  The  hostile  boats  may  approach  from  north  or  south.  In  this  case  it 
will  not  be  very  important  whether  the  OPV  is  the  leading  ship. 

5.  Interaction  of  the  Factors 

Interaction  of  the  terms  in  a  model  is  also  important  both  for  the  analyst 
and  the  decision  maker.  The  interaction  plot  for  the  second  order  model  is  shown 
in  Figure  34. 


Figure  34.  Interaction  plot  for  the  second  order  model. 


The  interaction  between  range  to  start  counter  measures  and  the 
presence  of  the  helicopter  seem  to  be  the  most  significant.  If  the  range  to  start 
counter  measure  is  high,  the  helicopter’s  presence  has  less  effect  on  the  mission 
success.  If  the  OPV  cannot  start  firing  early,  then  the  helicopter  becomes  more 
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important.  Therefore,  in  the  areas  where  there  is  significant  maritime  traffic,  the 
OTC  or  CO  should  consider  using  the  helicopter. 

The  presence  of  the  main  gun  appears  in  interactions  with  both  the 
auxiliary  gun’s  hit  probability  and  the  leading  ship  choice.  When  the  main  gun  is 
enabled,  the  hit  probability  of  the  auxiliary  gun  becomes  less  important.  Similarly, 
when  the  leading  ship  is  the  OPV,  whether  or  not  the  main  gun  is  enabled  has 
less  effect  on  the  outcome. 

The  main  insight  gained  from  interactions  is  that  theydemonstrate  a  simple 
way  to  perform  trade  space  analysis  between  the  choices.  For  instance,  if  the 
main  gun  is  malfunctioning,  the  decision  maker  should  consider  putting  the  OPV 
in  the  front  and  making  the  other  guns  more  effective. 

6.  Insignificant  Factors 

Last,  but  not  least,  having  a  significant  factor  which  appears  in  each  model 
is  important;  however,  it  is  almost  equally  important  not  seeing  a  factor  in  the 
significant  factors  list.  That  means  the  decision  maker  can  be  more  flexible  in  his 
or  her  decisions  for  that  specific  capability  or  tactic. 

Moreover,  if  a  factor  is  not  important  in  the  presence  of  others,  this  does 
not  mean  that  it  is  not  important  by  itself.  For  instance,  the  probability  of  hit  for 
the  main  gun  did  not  show  up  as  a  significant  factor.  However,  if  the  auxiliary  gun 
is  somehow  in  non-working  condition,  the  hit  probability  of  the  main  gun  becomes 
more  important.  A  trade  space  analysis  is  therefore  crucial  in  the  early  stages  of 
the  acquisition  process. 

G.  OTHER  IMPLEMENTATIONS  OF  THE  COMBAT  MODEL 

The  scenario  built  for  this  thesis  was  also  used  in  two  other  ways.  First,  it 
was  used  to  test  the  new  version  of  MANA.  An  experiment  on  both  versions 
showed  that  the  results  were  not  significantly  different  than  the  results  of  the 
previous  version  of  MANA. 
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Following  this  experiment,  more  factors  were  added  to  the  design  such 
that  the  number  of  factors  to  be  tested  was  increased  from  35  to  65.  These 
factors  were  changed  by  very  small  amounts,  and  so  were  not  expected  to  have 
important  impacts  on  the  mission  success.  Rather  than  running  a  new  512- 
design-point  experiment,  a  new  Lasso-optimal  screening  design  was  replicated 
1000  times,  and  the  resulting  data  were  analyzed.  The  supersaturated  screening 
design  used  in  this  method  uses  only  two  levels  for  each  factor  and  can  examine 
up  to  69  factors  with  only  24  design  points.  The  results  revealed  that  some  of  the 
design  points  cause  the  simulation  to  crash.  Building  the  partition  tree  shown  in 
Figure  35  demonstrated  that  in  some  cases  MANA  could  not  complete  the 
simulation  for  this  specific  scenario.  The  problem  only  occurs  when  Squad3  (the 
helicopter)  is  active,  and  when  a  specific  personality  value  of  this  agent  is  non¬ 
zero.  These  surprising  results  show  how  a  well-designed  screening  experiment 
can  be  a  useful  and  efficient  approach  to  model  verification:  it  would  have  been 
much  more  difficult  to  identify  the  software  problem  using  a  trial-and-error 
approach. 


Figure  35.  Partition  tree  to  show  the  factor  which  causes  MANA  to  crash. 
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Finally,  the  problematic  factor  was  removed  from  the  study  and  replaced 
with  another  in  order  to  test  the  analysis  capabilities  of  a  newly  proposed  Lasso- 
optimal  analysis  method  for  this  screening  design  (Xing,  Wan,  Zhu,  Sanchez,  & 
Kaymal,  2013).  The  purpose  of  this  effort  is  to  identify  significant  factors  in  a 
simulation.  Lasso  is  a  popular  tool  in  data  mining.  It  is  mainly  useful  when  the 
number  of  factors  is  large,  and  only  a  small  subset  of  those  factors  is  important. 
Therefore,  it  has  been  proposed  that  Lasso  can  be  used  to  design  and  analyze 
screening  experiments,  even  though  it  is  known  to  give  biased  estimates.  The 
results  were  mixed  for  this  ASUW  scenario:  the  Lasso  model  included  several 
factors  that  were  not  identified  in  the  larger  design  and  omitted  several  that  were. 
Further  research  is  underway  to  see  whether  the  existence  of  important 
interactions  confuses  model  identification  for  Lasso,  or  whether  some  of  the 
additional  factors  are  truly  important. 
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V.  CONCLUSION 


A.  OVERVIEW 

This  research  investigated  the  impacts  of  capability  and  operational 
related  decisions  to  mission  success  of  surface  combatants  in  an  ASUW 
environment.  A  scenario  was  built  based  on  a  real  world  incident,  and  it  was 
implemented  in  the  combat  modeling  platform  MANA.  102400  simulations  were 
performed  in  total,  and  the  output  of  the  simulations  was  analyzed  using  a  variety 
of  statistical  methods.  The  results  of  the  analysis  yielded  important  insights  for 
the  decision  makers.  Both  the  tactical  decisions  and  OPV  capabilities  were 
proved  to  be  important  for  mission  success. 

B.  RECOMMENDATIONS 

The  ASUW  scenario  analyzed  in  this  study  provides  valuable  insights 
regarding  the  naval  shipbuilding  process,  employing  tactics,  and  ROE.  The 
following  recommendations  are  based  on  the  insights  gained  from  the  results  of 
the  analysis: 

•  The  acquisition  of  the  naval  ships  needs  thorough  analysis 
regarding  capability,  cost,  and  operational  effectiveness. 

•  The  decision  makers  of  the  shipbuilding  process,  and  the  decision 
makers  at  sea  (i.e.,  the  naval  officers)  should  work  together  to  set 
the  requirements  for  a  ship  design. 

•  Trade-space  analysis  is  a  valuable  tool  to  understand  the  impact  of 
the  various  factors. 

•  Many  factors  affect  the  result  of  a  naval  mission.  Among  these, 
there  is  not  “one”  set  of  decisions  that  provides  the  mission 
success.  There  are  numerous  ways  to  achieve  the  same  result. 

•  Bigger  scale  modeling  and  simulations  can  be  performed  to 

analyze  the  operational  effectiveness  of  the  naval  ships  in  various 
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tasks.  This  will  help  the  planning  of  the  CONOPS  along  with  the 
shipbuilding  process. 

These  recommendations  represent  the  general  idea  of  the  implications  of 
this  research.  There  are  many  more  insights  that  can  be  gained  by  the  reader. 

C.  FURTHER  RESEARCH 

This  research  analyzed  only  one  ASUW  scenario  that  a  naval  ship  may 
face.  Building  other  scenarios  for  ASUW  and  for  other  warfare  areas  may 
improve  the  NPS  Dashboard.  This  fine-tuning  will  help  the  decision  makers  better 
understand  the  trade-space  between  factors  affecting  a  naval  mission.  Another 
important  study  that  can  be  done  is  building  a  combat  model  that  incorporates 
several  warfare  areas  such  as  ASUW,  ASW,  and  AAW  at  the  same  time. 

For  the  researcher  who  wants  to  expand  the  scenario  studied  in  this 
thesis,  the  following  improvements  can  be  added  to  the  combat  model  in  order  to 
make  a  more  thorough  analysis: 

•  Setting  aperture  angles  for  the  weapons  and  sensors 

•  Increasing  the  number  of  OPVs 

•  Using  several  UAVs  instead  of  a  helicopter 

•  Allowing  terrorists  to  carry  weapons  such  as  rockets. 

D.  SUMMARY 

In  the  past,  the  experience  of  the  sailors  was  the  main  driver  of  the  tactical 
decisions.  Today,  it  is  the  technology  and  the  science  that  shape  tactical  decision 
making  more  intensely.  Naval  officers  should  know  the  characteristics  of  their 
ships,  and  they  should  rely  not  only  their  experience  but  also  on  analytical  tools 
to  form  their  decisions.  Choosing  the  right  course  of  action  may  reduce  the 
emphasis  on  capability  as  the  primary  factor  in  decision  making.  This  change  has 
the  potential  to  reduce  the  acquisition  cost  of  a  naval  ship  while  improving  the 
navy’s  prospects  for  achieving  mission  success. 
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